Simple Hand Tracking with kinect + opencv

in Gallery • 9 years ago

Wanted to do something really basic just to get a handle on blob tracking.

#include "cinder/app/AppBasic.h"
#include "cinder/gl/gl.h"
#include "cinder/gl/Texture.h"
#include "cinder/params/Params.h"
#include "Kinect.h"
#include "CinderOpenCv.h"
using namespace ci;
using namespace ci::app;
using namespace std;
class HandTrackingApp : public AppBasic {
public:
void prepareSettings( Settings* settings );
void setup();
void update();
void draw();
params::InterfaceGl mParams;
float mThreshold, mBlobMin, mBlobMax;
float mKinectTilt;
Kinect mKinect;
gl::Texture mColorTexture, mDepthTexture, mCvTexture;
Surface mDepthSurface;
Vec3f mTargetPosition;
};
void HandTrackingApp::prepareSettings( Settings* settings )
{
settings->setWindowSize( 640, 720 );
}
void HandTrackingApp::setup()
{
mThreshold = 70.0f;
mBlobMin = 20.0f;
mBlobMax = 80.0f;
mParams = params::InterfaceGl( "Hand Tracking", Vec2i( 10, 10 ) );
mParams.addParam( "Threshold", &mThreshold, "min=0.0 max=255.0 step=1.0 keyIncr=s keyDecr=w" );
mParams.addParam( "Blob Minimum Radius", &mBlobMin, "min=1.0 max=200.0 step=1.0 keyIncr=e keyDecr=d" );
mParams.addParam( "Blob Maximum Radius", &mBlobMax, "min=1.0 max=200.0 step=1.0 keyIncr=r keyDecr=f" );
mParams.addParam( "Kinect Tilt", &mKinectTilt, "min=-31 max=31 keyIncr=T keyDecr=t" );
mKinect = Kinect( Kinect::Device() );
mTargetPosition = Vec3f::zero();
}
void HandTrackingApp::update()
{
if( mKinect.checkNewDepthFrame() ){
ImageSourceRef depthImage = mKinect.getDepthImage();
// make a texture to display
mDepthTexture = depthImage;
// make a surface for opencv
mDepthSurface = depthImage;
if(mDepthSurface){
// once the surface is avalable pass it to opencv
// had trouble here with bit depth. surface comes in full color, needed to crush it down
cv::Mat input( toOcv( Channel8u( mDepthSurface ) ) ), blurred, thresholded, thresholded2, output;
cv::blur(input, blurred, cv::Size(10,10));
// make two thresholded images one to display and one
// to pass to find contours since its process alters the image
cv::threshold( blurred, thresholded, mThreshold, 255, CV_THRESH_BINARY);
cv::threshold( blurred, thresholded2, mThreshold, 255, CV_THRESH_BINARY);
// 2d vector to store the found contours
vector<vector<cv::Point> > contours;
// find em
cv::findContours(thresholded, contours, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE);
// convert theshold image to color for output
// so we can draw blobs on it
cv::cvtColor( thresholded2, output, CV_GRAY2RGB );
// loop the stored contours
for (vector<vector<cv::Point> >::iterator it=contours.begin() ; it < contours.end(); it++ ){
// center abd radius for current blob
cv::Point2f center;
float radius;
// convert the cuntour point to a matrix
vector<cv::Point> pts = *it;
cv::Mat pointsMatrix = cv::Mat(pts);
// pass to min enclosing circle to make the blob
cv::minEnclosingCircle(pointsMatrix, center, radius);
cv::Scalar color( 0, 255, 0 );
if (radius > mBlobMin && radius < mBlobMax) {
// draw the blob if it's in range
cv::circle(output, center, radius, color);
//update the target position
mTargetPosition.x = 640 - center.x;
mTargetPosition.y = center.y;
mTargetPosition.z = 0;
}
}
mCvTexture = gl::Texture( fromOcv( output ) );
}
}
if( mKinect.checkNewColorFrame() )
mColorTexture = mKinect.getColorImage();
if( mKinectTilt != mKinect.getTilt() )
mKinect.setTilt( mKinectTilt );
}
void HandTrackingApp::draw()
{
gl::clear( Color( 0.5f, 0.5f, 0.5f ) );
gl::disableDepthWrite();
gl::disableDepthRead();
glPushMatrix();
gl::scale(Vec3f(-1, 1, 1));
if( mColorTexture )
gl::draw( mColorTexture, Vec2i( -640, 0));
glPopMatrix();
glPushMatrix();
gl::scale(Vec3f(-0.5, 0.5, 1));
if( mDepthTexture )
gl::draw( mDepthTexture,Vec2i( -640, 920 ));
if ( mCvTexture )
gl::draw( mCvTexture,Vec2i( -1280, 920 ));
glPopMatrix();
gl::enableDepthWrite();
gl::enableDepthRead();
gl::color(Colorf(1.0f, 0.0f, 0.0f));
gl::drawSphere(mTargetPosition, 10.0f);
params::InterfaceGl::draw();
}
CINDER_APP_BASIC( HandTrackingApp, RendererGl )

Thats the meat of it. Thanks to letsgooutside, sansumbrella, and a.stellato for related posts in the forum that helped me get over some humps.

Replies(32)

nardove

Re: Simple Hand Tracking with kinect + opencv

9 years ago

Hi Andrew,

Thanks a lot for sharing, this will be great to get me started learning opencv

- rS

flight404

Re: Simple Hand Tracking with kinect + opencv

9 years ago

Just gave it a go and it works like a charm. And as nardove said, a good way to get a foot in the OpenCV door. Thanks Andy.

BTW, I see you were trying to adjust the size of the Params window and based on the 10, 10, I am guessing you were wondering why it wasn't resizing. Turns out if the name of the Params window has a space in it, the resize wont work. :/ So change it to "HandTracking" and the size will start working.

tombeddard

Re: Simple Hand Tracking with kinect + opencv

9 years ago

Thanks for sharing this Andrew. Strangely though I'm getting errors when linking in the build process in Xcode (the opencv and kinect block examples are working fine):

Any suggestions?

Thanks

sansumbre..

Re: Re: Simple Hand Tracking with kinect + opencv

9 years ago

Make sure the path to your opencv library ( libcv.a libcxcore.a libml.a ) is correct in your "Other Linker Flags" The error shows that the headers are found, but the compile library isn't. Andrew Bell's post on using cinder blocks points you in the right direction for adding linker flags.

reddoorso..

Re: Simple Hand Tracking with kinect + opencv

9 years ago

Thanks for this. I think I have followed everything that I can think of setting the project (header/linker). But now I get these errors.

Build HandTracking of project HandTracking with configuration Debug

Ld build/Debug/HandTracking.app/Contents/MacOS/HandTracking normal i386

cd /Users/MB/Documents/Code/cinder/MyCinderCodes/HandTracking/xcode

setenv MACOSX_DEPLOYMENT_TARGET 10.5

/Developer/usr/bin/g++-4.2 -arch i386 -isysroot /Developer/SDKs/MacOSX10.6.sdk -L/Users/MB/Documents/Code/cinder/MyCinderCodes/HandTracking/xcode/build/Debug -F/Users/MB/Documents/Code/cinder/MyCinderCodes/HandTracking/xcode/build/Debug -filelist /Users/MB/Documents/Code/cinder/MyCinderCodes/HandTracking/xcode/build/HandTracking.build/Debug/HandTracking.build/Objects-normal/i386/HandTracking.LinkFileList -mmacosx-version-min=10.5 ../../../lib/libcinder_d.a ../../../blocks/opencv/lib/macosx/libcv.a ../../../blocks/opencv/lib/macosx/libml.a ../../../blocks/opencv/lib/macosx/libcxcore.a ../../../blocks/Cinder-Kinect/lib/macosx/libfreenect.a ../../../blocks/Cinder-Kinect/lib/macosx/libusb-1.0.a -framework Cocoa -framework OpenGL -framework Carbon -framework CoreVideo -framework QTKit -framework QuickTime -framework Accelerate -framework AudioToolbox -framework AudioUnit -framework CoreAudio -o /Users/MB/Documents/Code/cinder/MyCinderCodes/HandTracking/xcode/build/Debug/HandTracking.app/Contents/MacOS/HandTracking

ld: warning: cv::Exception::what() consthas different visibility (default) in ../../../blocks/opencv/lib/macosx/libcxcore.a(cxsystem.o) and (hidden) in /Users/MB/Documents/Code/cinder/MyCinderCodes/HandTracking/xcode/build/HandTracking.build/Debug/HandTracking.build/Objects-normal/i386/HandTrackingApp.o

ld: warning: cv::Exception::~Exception()has different visibility (default) in ../../../blocks/opencv/lib/macosx/libcxcore.a(cxsystem.o) and (hidden) in /Users/MB/Documents/Code/cinder/MyCinderCodes/HandTracking/xcode/build/HandTracking.build/Debug/HandTracking.build/Objects-normal/i386/HandTrackingApp.o

ld: warning: cv::Exception::~Exception()has different visibility (default) in ../../../blocks/opencv/lib/macosx/libcxcore.a(cxsystem.o) and (hidden) in /Users/MB/Documents/Code/cinder/MyCinderCodes/HandTracking/xcode/build/HandTracking.build/Debug/HandTracking.build/Objects-normal/i386/HandTrackingApp.o

ld: warning: cv::Mat::create(int, int, int)has different visibility (default) in ../../../blocks/opencv/lib/macosx/libcv.a(cvthresh.o) and (hidden) in /Users/MB/Documents/Code/cinder/MyCinderCodes/HandTracking/xcode/build/HandTracking.build/Debug/HandTracking.build/Objects-normal/i386/HandTrackingApp.o

ld: warning: cv::Mat::release() has different visibility (default) in ../../../blocks/opencv/lib/macosx/libcv.a(cvfilter.o) and (hidden) in /Users/MB/Documents/Code/cinder/MyCinderCodes/HandTracking/xcode/build/HandTracking.build/Debug/HandTracking.build/Objects-normal/i386/HandTrackingApp.o

ld: warning: std::vector<cv::Point_<int>, std::allocator<cv::Point_<int> > >::~vector()has different visibility (default) in ../../../blocks/opencv/lib/macosx/libcv.a(cvfilter.o) and (hidden) in /Users/MB/Documents/Code/cinder/MyCinderCodes/HandTracking/xcode/build/HandTracking.build/Debug/HandTracking.build/Objects-normal/i386/HandTrackingApp.o

ld: warning: cv::Exception::formatMessage() has different visibility (default) in ../../../blocks/opencv/lib/macosx/libcv.a(cvshapedescr.o) and (hidden) in /Users/MB/Documents/Code/cinder/MyCinderCodes/HandTracking/xcode/build/HandTracking.build/Debug/HandTracking.build/Objects-normal/i386/HandTrackingApp.o

Undefined symbols:

"_IONotificationPortGetRunLoopSource", referenced from:

_event_thread_main in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

"_IORegistryEntryCreateCFProperty", referenced from:

_darwin_kernel_driver_active in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

_darwin_devices_detached in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

"_IOServiceAddMatchingNotification", referenced from:

_event_thread_main in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

"_IODestroyPlugInInterface", referenced from:

_usb_get_next_device in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

"_IOServiceMatching", referenced from:

_usb_setup_device_iterator in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

_event_thread_main in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

"_IONotificationPortCreate", referenced from:

_event_thread_main in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

"_IOIteratorIsValid", referenced from:

_usb_get_next_device in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

"_IOIteratorNext", referenced from:

_darwin_get_interface in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

_usb_get_next_device in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

_darwin_get_device_list in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

_event_thread_main in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

_darwin_devices_detached in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

"_IOServiceGetMatchingServices", referenced from:

_usb_setup_device_iterator in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

"_IOObjectRelease", referenced from:

_darwin_get_interface in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

_darwin_get_interface in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

_darwin_kernel_driver_active in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

_darwin_claim_interface in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

_usb_get_next_device in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

_darwin_get_device in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

_darwin_get_device_list in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

_darwin_get_device_list in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

_darwin_get_device_list in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

_event_thread_main in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

_darwin_devices_detached in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

"_IOCreatePlugInInterfaceForService", referenced from:

_darwin_claim_interface in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

_usb_get_next_device in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

"_IOMasterPort", referenced from:

_darwin_init in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

"_IONotificationPortDestroy", referenced from:

_event_thread_main in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

ld: symbol(s) not found

collect2: ld returned 1 exit status

Interestingly, if I copy the code to one of the Kinect sample projects, it works. But if I try to build a project from the scratch using TinderBox, it will not work.

I am at my wits' end.

Any help?

sansumbre..

Re: Re: Simple Hand Tracking with kinect + opencv

9 years ago

Looks like you also need to include the libusb library. Check that you're including all the kinect binaries in your other linker flags. Good luck and Merry Christmas. I don't have a kinect myself, but it looks like you just need to keep including the right libs to get it working at this point.

reddoorso..

Re: Simple Hand Tracking with kinect + opencv

9 years ago

Thanks, Sansumbrella.

But it's not working. I really want to know what I am missing here.

This is the process that I kept repeating.

1. Create a new cinder project (named: HandTracking) using TinderBox in the same folder where the two Kinect samples are.

2. Copy both the Project setting (header) and Target setting (other linker flags) from the one that is working (for this, I used the "KinectBasic" project and changed the project/target settings and added necessary codes) to the new cinder project.

3. Add Kinect.app to the source folder.

4. Copy the exact code from HandTrackingApp.cpp that is working to the new project.

I compiled it and got the exact same errors above. What am I missing??

Should I suspect that a new project created by TinderBox has some different settings other than headers/linker flags that I should know about?

cschwesig

Re: Re: Simple Hand Tracking with kinect + opencv

9 years ago

FWIW, I'm getting exactly the same errors following the same steps, although I've managed to use both Cinder-Kinect and opencv in another project.

I'm new to Cinder, but doesn't the following error mean that libusb-1.0.a is included, but fails to find "_IONotificationPortGetRunLoopSource"?

"_IONotificationPortGetRunLoopSource", referenced from:

_event_thread_main in libusb-1.0.a(libusb_1_0_la-darwin_usb.o)

reddoorso..

Re: Simple Hand Tracking with kinect + opencv

9 years ago

Solved!

The new project created by TinderBox does not include IOKit.Framework. As soon as I included it to my project, it compiled and worked perfectly. I double-checked this by creating a dummy project with TinderBox. Again, the dummy project did not include IOKit.Framework.

Is the reason why TinderBox does not include IOKit.Framework by default is because it is not usually required for most of the projects?

subblue

Re: Re: Simple Hand Tracking with kinect + opencv

9 years ago

Thanks. IOKit.framework was what I was missing from my Tinderbox created project too. All working now :)

crisegs

Re: Simple Hand Tracking with kinect + opencv

9 years ago

hi,
i need help

i have 15 errors

emmanuel.g

Réf.: Simple Hand Tracking with kinect + opencv

9 years ago

Hey Andy,

Thank you very much for providing this sample!

I've succeeded to compile your code, but it seems that I don't get the upper video frame...

check my screenshot?

any clue why I only get the two bottom frame but not the main one?

Thank you!

emmanuel

dazzid

Re: Simple Hand Tracking with kinect + opencv

9 years ago

Hi All,

First of all thanks for sharing this!

I had this example running well in Xcode 3. Now with Xcode 4 I'm facing many issues.

I remake the all project and now I'm not sure how to fix it, and after many hours I think it is time to ask for help:

This is how I'm organizing my project, I did the project using TinderBox. The error I have is a lot bigger than my knowledge.

Any suggestion? (not only to read "Accelerated C++" book)

Thanks for the help!

David

dazzid

Re: Re: Simple Hand Tracking with kinect + opencv

9 years ago

Ok I've found the problem. The compiler version is GCC 4.2 and not LLVM compiler 2.0. Not sure why and I don't know about this compilers yet, but I took me two coffees and one day to catch it... that's the way things are sometimes.

miggsb

Re: Simple Hand Tracking with kinect + opencv

9 years ago

I'm get the same problems as crisegs.

I copied the opencv block from Cinder 0.8.2 to the blocks folder in version of Cinder I built from git
The opencv examples build fine when I use 0.8.2 but none of them build for the git version of Cinder.
libcv.a, libcxcore.a, and libml.a have been added according to Andrew's "how to add cinder blocks to xcode"
Kinect examples compile fine.
Using xcode 3.2.5 on os x 10.6.7
The errors are coming from "CinderOpenCV.h" :

Is there something I should know about using opencv with the latest cinder? The project I'm working requires both opencv and a kinect so that rules out using 0.8.2 right?

I should add I'm new to Cinder.

Thanks I've been struggling with this all morning.

andrewfb

Re: Simple Hand Tracking with kinect + opencv

9 years ago

Hey - I think you'll need to pull the newer Cinder-OpenCV from git to match if you're using Cinder from git. You can

git clone git://github.com/cinder/Cinder-OpenCV.git

from your blocks folder to do so.

miggsb

Re: Simple Hand Tracking with kinect + opencv

9 years ago

Thanks Andrew, I just figured that out an came back to update my posting.

Wish I thought of it couple of hours ago :/

Thank you!

-Miguel

sooki81

Re: Simple Hand Tracking with kinect + opencv

8 years ago

Hi Andy

Thank you so much for this. I was hoping you can help with an error I keep receiving when I build:

Command /Developer/usr/bin/gcc-4.2 failed with exit code 1

Its driving me loopy!

Suki

jeanhochu

Re: Re: Simple Hand Tracking with kinect + opencv

8 years ago

I am having the same error with you. I was just using the kinect point cloud example. somebody please help me. thanks

zagor

R: Simple Hand Tracking with kinect + opencv

8 years ago

HI,

your code works perfectly even with BlockOpenNI. I have just a question, why everything is in red ?

I saw screenshots here in this thread and it seems they have the same problem, does somebody know why?

reagrds,

Luca

chonigman

Re: Simple Hand Tracking with kinect + opencv

8 years ago

Hey everybody,

So I just posted a second ago asking about the red tint as well. Five seconds later I figured it out.

At the bottom of the draw loop I changed

gl::color(Colorf(1.0f, 0.0f, 0.0f));

gl::color(Colorf(1.0f, 1.0f, 1.0f));

Also I was having trouble getting the top half of the window to display video. At first I was getting errors because of this line of code

if( mKinect.checkNewColorFrame() )

mColorTexture = mKinect.getColorImage();

I took a look at the Kinect.cpp and did not find anything called "checkNewColorFrame()" or "getColorImage()". So I went ahead and commented it out. This makes the whole top area gray except for the circle. So I took a look at the example Kinect for Cinder programs and saw that they use this code to get the regular camera view:

if( mKinect.checkNewVideoFrame() ){

mColorTexture = mKinect.getVideoImage();

}

And this solved the problem. Now everything looks great and I am now adding audio to the circle position.

Hope this helps anybody else who was stumped.

Thanks again,

Colin

Allan Yong

Re: Simple Hand Tracking with kinect + opencv

8 years ago

Also a quick note for anyone how has linking error like I did (resulting in a number of hours),

Remember to add Kinect.cpp file to your project's source folder...

coder.guy

Re: Simple Hand Tracking with kinect + opencv

8 years ago

Hello All,

Just curious to see what CPU utilization numbers people are getting. I've been playing with this over the weekend and seem to consistently get around 80% CPU utilization which I think. The generic MS kinect example tops out at 20% with skeleton tracking so i doubt that simply using depth w/openCV would warrant such a high CPU utilization. I even tried testing only every 20th frame to no avail on making the numbers drop. Not entirely sure if something is wrong on my end but just thought I'd get other people's experience with this.

Thanks,

CG

paul.houx

Reageren: Re: Simple Hand Tracking with kinect + opencv

8 years ago

Hi CG,

Cinder's own game loop might interfere with the results. What kind of CPU usage do you get when running the empty BasicApp sample?

Quite recently a few changes were made to Cinder that reduce CPU usage. See for more information:

http://forum.libcinder.org/topic/stable-frame-rate-and-low-cpu

-Paul

coder.guy

Re: Simple Hand Tracking with kinect + opencv

8 years ago

Paul,

Thanks for the information. The basicApp sample is running at 25% CPU. Is there a way to bring this down? I see the changes were committed and are in .83 which is what i am running on windows 7 using visual C++ express.

Thanks,

CG

paul.houx

Reageren: Re: Simple Hand Tracking with kinect + opencv

8 years ago

Hi CG,

the changes are only committed to the GitHub repository and are not part of the official v0.8.3 release. The BasicApp sample should run at (less than) 1% with the changes applied. It probably runs at (close to) 100% of a single CPU core without the changes.

If you have a quad core processor and BasicApp runs at 25%, it is probably using 1 core at 100%.

For detailed information about CPU usage:

Open Windows Task Manager (CTRL+Shift+Esc)
Click on the Performance tab
Click on 'Resource Monitor...'
Click on the CPU tab

Note: the value in the 'Average CPU' column is calculated over the last 60 seconds(!), which means you should run your application for at least a minute to get relatively accurate values there.

-Paul

coder.guy

Re: Simple Hand Tracking with kinect + opencv

8 years ago

Paul,

letsgooutside just responded to another issue with the newest version of the block that may have an impact. The block is untested in BootCamp so that may be the issue. If you have any tips on improving performance i can try please let me know. I'll be looking for a PC to run some more tests.

Thanks,

CG

Edit: Just saw your response Paul. I'll pull from git master to see if the changes impact performance. More to come.

shabanzo

Re: Simple Hand Tracking with kinect + opencv

6 years ago

hi,

how I can segmenting a solid blob like yours (bottom right window)? I've followed your code but my blob not solid like yours.

thanks. :)

Léonard

Réf.: Simple Hand Tracking with kinect + opencv

3 years ago

Hi,

Trying to get started with Kinect in Cinder here.

Basic question: where do you define the Kinect class? There's no sign of any Kinect.h around, although the code includes it. Was I supposed to do something special when creating the project from Tinderbox?

Thanks !

Léo.

paul.houx