Hi, makers, we developed a simple program which can use Kinect for Windows to control uArm through Visual Studio.
There are three main functions that the program can do:
1, 3D coordinates Control
2, Play piano
3, Calligraphy
Here are the videos below show 2 main functions:
Play Piano
Calligraphy
You can follow the tutorials below show what we have done for making these videos:
Step 1: Getting the Materials
- uArm Metal version or Acrylic version (view our official website)
- Microsoft Kinect for Windows
- Kinect SDK for Windows
- Visual Studio 2013
- *Optional – toy piano
- *Optional – Writing Brush and Pen Holder
Step 2: Getting Left-Hand Coordinates from Kinect
Our Goal is to control uArm only for left hand, so in the Visual Studio, we should acquire left Hand Coordinates.
1, First, in the opening function, define sensor
KinectSensor sensor = KinectSensor.GetDefault();
2, When sensor is opened, read each frame from Kinect by using
// Read both color image and body image from kinect sensor
reader = sensor.OpenMultiSourceFrameReader(FrameSourceTypes.Color | FrameSourceTypes.Body);
reader.MultiSourceFrameArrived += Reader_MultiSourceFrameArrived;
3, Then, in the body frame, define left hand that:
Joint leftHand = body.Joints[JointType.HandLeft];
4, After that, we can obtain the coordinates for left-hand:
// enlarge the value in meter
leftxvalue = leftHand.Position.X * 100;
leftyvalue = leftHand.Position.Y * 100;
leftzvalue = leftHand.Position.Z * 100;
5, Finally, read hand opened or closed status by using
if (body.HandLeftState == HandState.Closed) or (HandState.Open)
Step 3: Transfer and Process Data
After obtain data, there are two more things need to be done: process data that uArm can execute and transfer the data through serial communication
1, Process data
First we need to map the data from kinect to arduino coordinates.
Then, we need to use inverse-kinematics to generate the angles each servo should execute in uArm by adding calAngles(x, y, z) function
2, Transfer data
We transfer data by using Firmata-EEPROM, that, we send the data uArm can read and execute.
In the uArm side, just download UCP (communication protocol) to Arduino.
Step 4: Decide the control method
Before sending data to uArm, first we need to decide when should we send data and what types of data should be sent.
We made the control method that, when you close both of your hands for 2 seconds, Visual Studio will read the send data to uArm as the video below.
Above that, we also made the User-Interface that.
Step 5: Begin to Use
After all the settings and coding, we can control uArm through Kinect.
Here are the final pictures:
If you want to use our code directly, remember to download both Visual-studio project in your computer with kinect SDK 2.0.
And also download UCP (serial communication file) to uArm!!
All code are free to download, checkout this link to download the
Source code:
https://github.com/uArm-Developer/UArmForArduino
UArm For Arduino – Standard Firmata EEPROM:
https://github.com/uArm-Developer/uArmForArduino
Kinect SDK
http://www.microsoft.com/en-gb/download/details.aspx?id=44561
You can also check our Youtube channel to see what are we doing now~
https://www.youtube.com/channel/UCyy5ekYtq35jFtPpY3O_tVA/videos
- before playing music with uArm, remember you have calibrated uArm in the latest way, check this website below to calibrate uArm and use all new APIs .
developer.evol.net
Cheers
No Comments