Hi, makers, we developed a simple program which can use Kinect for Windows to control uArm through Visual Studio.

There are three main functions that the program can do:

1, 3D coordinates Control

2, Play piano

3, Calligraphy 

 

Here are the videos below show 2 main functions:

Play Piano

Calligraphy

 

You can follow the tutorials below show what we have done for making these videos:

 

Step 1: Getting the Materials 

• uArm Metal version or Acrylic version (view our official website)
• Microsoft Kinect for Windows
• Kinect SDK for Windows
• Visual Studio 2013
• *Optional – toy piano
• *Optional – Writing Brush and Pen Holder

 

Step 2: Getting Left-Hand Coordinates from Kinect

Our Goal is to control uArm only for left hand, so in the Visual Studio, we should acquire left Hand Coordinates.

1, First, in the opening function, define sensor

      KinectSensor sensor = KinectSensor.GetDefault();

2, When sensor is opened, read each frame from Kinect by using

      // Read both color image and body image from kinect sensor
      reader = sensor.OpenMultiSourceFrameReader(FrameSourceTypes.Color | FrameSourceTypes.Body);
      reader.MultiSourceFrameArrived += Reader_MultiSourceFrameArrived;

3, Then, in the body frame, define left hand that:

       Joint leftHand = body.Joints[JointType.HandLeft];

4, After that, we can obtain the coordinates for left-hand:

 

      // enlarge the value in meter

      leftxvalue = leftHand.Position.X * 100;
      leftyvalue = leftHand.Position.Y * 100;
      leftzvalue = leftHand.Position.Z * 100;

 

5, Finally, read hand opened or closed status by using

if (body.HandLeftState == HandState.Closed) or (HandState.Open)

 

Step 3: Transfer and Process Data

After obtain data, there are two more things need to be done: process data that uArm can execute and transfer the data through serial communication

1, Process data

First we need to map the data from kinect to arduino coordinates.

Then, we need to use inverse-kinematics to generate the angles each servo should execute in uArm by adding  calAngles(x, y, z) function

 

2, Transfer data

We transfer data by using Firmata-EEPROM, that, we send the data uArm can read and execute.

In the uArm side, just download UCP (communication protocol) to Arduino.

 

 

Step 4: Decide the control method

Before sending data to uArm, first we need to decide when should we send data and what types of data should be sent.

We made the control method that, when you close both of your hands for 2 seconds, Visual Studio will read the send data to uArm as the video below.

 

Above that, we also made the User-Interface that.

 

Step 5: Begin to Use

 

After all the settings and coding, we can control uArm through Kinect.

Here are the final pictures:

 

If you want to use our code directly, remember to download both Visual-studio project in your computer with kinect SDK 2.0.

And also download UCP (serial communication file) to uArm!!

 

All  code are free to download, checkout this link to download the

Source code: 

https://github.com/uArm-Developer/UArmForArduino

UArm For Arduino – Standard Firmata EEPROM:

https://github.com/uArm-Developer/uArmForArduino

Kinect SDK

http://www.microsoft.com/en-gb/download/details.aspx?id=44561

 

You can also check our Youtube channel to see what are we doing now~

https://www.youtube.com/channel/UCyy5ekYtq35jFtPpY3O_tVA/videos

• before playing music with uArm, remember you have calibrated uArm in the latest way, check this website below to calibrate uArm and use all new APIs .

developer.evol.net

 

Cheers