Kinect SDK block

Developing Cinder

letsgoout..

in Developing Cinder • 1 year ago

The KinectSdk block has grown up. Read all about it here:

http://www.bantherewind.com/kinectsdk-for-cinder

Get it here:

https://github.com/BanTheRewind/Cinder-KinectSdk

New "PointCloudGpu" sample app demos GPU-based point cloud and the "camera follow" technique

New "KinectApp" leverages additions to the SDK wrapper

Replies(126)

makemachi..

Re: Kinect SDK block

1 year ago

Steve, this is rad. Nice job!

This is probably a long shot, but I'm running a Mac and considering installing Windows 7 & Visual Studio through Bootcamp in order to make this work. If anyone out there has a similar configuration and gets this running please let the list know! I've only found one thread regarding this topic, and it appears that an issue arose where the graphics card on the Mac may not have supported a DirectX setting. It seems like it may not be a problem if the rendering was through OpenGL. Wishful thinking?

letsgoout..

Re: Kinect SDK block

1 year ago

I'd say give it a shot. The SDK's examples use DirectX, but this one obviously does not. I'm also using boost::thread instead of CreateThread now, and as few as possible Windows-specific type names (eg, DWORD, HANDLE, etc). Basically, the only Windows-specific stuff I'm using are the actual Kinect API calls.

Point being that you'll have *far* less Windows-in-Bootcamp issues to tackle than what's in the official demo projects. Worth a try, IMO.

http://www.BanTheRewind.com

ian.jones..

Re: Kinect SDK block

1 year ago

Silly question: Is this Windows only?

pixelnerve

Re: Kinect SDK block

1 year ago

One thing. Kinect sdk only supports one or two users max. Not six.

letsgoout..

Re: Kinect SDK block

1 year ago

Actually, it does track six users. It only tracks two *skeletons*. You can stand six people in front of the Kinect and if you look at the depth image, six different colored people show up. However, only the last two skeletons appear.

http://www.BanTheRewind.com

a.stellato

Re: Kinect SDK block

1 year ago

Just got to check this out. Having done quite a lot of work with OpenNI, I am very excited to see the SDK acquire skeletons without any configuration pose. Really exciting stuff.

I too am experiencing the insert bug crash occasionally. It seems that your thread sync hypothesis might be true. Locks will probably solve the issue.

Great work on this project! Can't wait to have full Kinect support in Cinder!

- a.stellato

Beginning the bunny apocalypse.

letsgoout..

Re: Kinect SDK block

1 year ago

Been sick and out of commission since Saturday so sorry for the late reply.

@ian.jones.oz

Yes, the SDK is Windows-only, therefore the block is Windows-only.

@pixelnerve

You are correct. The SDK is a bit misleading in making it appear that you can cycle through six users. It *tracks* six users at a time, but only reports up to two. If users 1, 2, and 3 appear in front of the Kinect, 1 and 2 will show up. If 1 leaves the view, 2 and 3 will be on screen. If then 2 and 1 swap, 1 and 3 will show up (as opposed to 3 and 4). I didn't realize this until we actually had five people standing in front of the Kinect.

@a.stellato

It's 100% a thread sync issue. On Friday night, I switched the threading over to boost with locking. However, boost::thread does not pause when you call WaitForMultipleObjects, which only works inside of MS's CreateThread callback. This means the events are never triggered.

I'm looking into doing thread synchronization with CreateThread. Failing that, it might make sense to switch to an event-based system. I did this on my audioInput class for Windows for exactly that reason. I'd like to avoid that here only because the current class is so simple.

EDIT: I don't have a Kinect here, but I believe I've fixed the thread sync issue. Locking a Windows thread is actually very easy. Will post results tomorrow.

http://www.BanTheRewind.com

a.stellato

Re: Kinect SDK block

1 year ago

Really great that there are no more crashes, but now these examples run very, very slowly.

For the basic sample on my test machine, I used to get a fps of ~60. Now it runs at ~14 fps.

The point cloud example runs exceptionally slow, maybe at ~6 fps.

Not sure if you're experiencing the same degradation of performance, but it is very apparent on this setup.

- a.stellato

letsgoout..

Re: Kinect SDK block

1 year ago

I'm getting 60fps on both (make sure you're in release mode, not debug), but I added a few optimizations which might help. 0.0.3 beta is now in the original post or here:

http://bantherewind.com/kinect-sdk-block-for-cinder

Threads are now run inside a boost::thread_group which should better manage running threads on separate cores
You now have the ability to toggle tracking features, which gives a big performance boost when you turn off what you don't need:

Kinect::enableDepth(bool enable = true)
Kinect::enableSkeletons(bool enable = true)
Kinect::enableVideo(bool enable = true)

The "KinectApp" example has new parameters which allow you toggle features
The "PointCloudApp" example shuts off video and skeletons by default

I noticed that CPU usage drops by about 15% (of the total) for each feature you turn off. That is, something using 30% CPU tracking all features will use ~20% tracking just one. I can also definitely see the usage being spread across cores in the Task manager now.

http://www.BanTheRewind.com

makemachi..

Re: Kinect SDK block

1 year ago

So, I took the jump and installed Windows on my Mac to see if the Kinect SDK would run. I'm happy to report that it was a fairly painless process, with no real black magic involved.

Just install Windows 7 through Bootcamp, then install all of the dependencies and you'll be set.

There seem to be a couple of issues with the Cinder block, though I haven't done any debugging just yet. Neither example will run in debug mode. They run fine in release mode but do crash as the program exits. Both examples perform quite well, running at a consistent 60fps.

letsgoout..

Re: Kinect SDK block

1 year ago

Just added a snazzy new feature which lets you turn the depth image into a binary (black/white) image. You can invert it, as well. See the image below. This is handy for tracking user outlines as shown here with OpenNI + OpenCV.

As it turns out, the Kinect SDK *does* track six users simultaneously, so you can get clean outlines for up to six folks at a time. The two user limit only applies to skeletons.

http://www.BanTheRewind.com

justinmau..

Re: Kinect SDK block

1 year ago

I've also used your example and used the OSC cinder block to send out all the joint data for each skeleton tracked. It's been useful for others who don't know cinder or are working with other languages and frameworks to get going easily. Let me know if you're interested and I can send it over. Perhaps this block is deserving of a github account :).

letsgoout..

Re: Kinect SDK block

1 year ago

Nice. That's a great idea. I can see combining the current sample app with your work to build out something like CCV for Kinect.

I also wrote an app for recording and playing back skeleton positions to do motion tracking for animation. I could see combining these (ie, save the positions in XML as OSC formatted data).

I still haven't gotten my Git skills up to date since SVN is integrated into our project management software (hey, it works great!). But it might be time to dive in...

http://www.BanTheRewind.com

justinmau..

Re: Kinect SDK block

1 year ago

pm'd you letsgooutside with my additions!

letsgoout..

Re: Kinect SDK block

1 year ago

Awesome. I moved things around just a tad so that it uses the OSC block that comes with Cinder instead of looking at a "local" copy. But other than that it works right out of the box. A co-worker just installed the official drivers on his machine and the application worked right away (phew). He's building a test bed in Flash right now. :)

http://www.BanTheRewind.com

justinmau..

Re: Kinect SDK block

1 year ago

Yea - that makes more sense. Glad to hear you got it working! If you wanted to merge it with your more current work you should just be able to look for any lines that have @JustinMaurer and copy the corresponding chunk of code below. Otherwise I'll post it here so others can start using it. I just didn't want a lot of versions confusing everyone. :)

letsgoout..

Re: Kinect SDK block

1 year ago

0.0.5 beta is up now. It includes a getUserCount() method which reports the number of users based on the depth image (ie, not skeletons). This is handy for skipping over processing when it isn't needed or detecting inactivity.

The sample application now reports the number of users in the parameters.

The user count will always be zero if you if depth is disabled. The number of skeletons is more obvious and easy to check on your own, so there is no need for a convenience method there.

This should be the last bit of development on the block for awhile, as I think I have all the features in and they're working well. Now to move onto the OSC dispatcher app. :)

PS

We've got this working with Flash over OSC here. :D

http://www.BanTheRewind.com

letsgoout..

Re: Kinect SDK block

1 year ago

OK, final update before going on Github for reals. 0.0.6b includes a few code optimizations (namely moving using namespace from *.h into *.cpp files) and a new sample project. "SkeletonApp" is a stripped down version of "KinectApp" which only demonstrates acquiring and rendering skeletons for those who need nothing more.

http://www.BanTheRewind.com

letsgoout..

Re: Kinect SDK block

1 year ago

Russ here at Fashionbuddha is working on a test bed to get Kinect data in Flash. The results are fantastic. There is no real noticeable lag between the Cinder app and the Flash app. It's super responsive. Great work, Justin!

http://www.BanTheRewind.com

letsgoout..

Re: Kinect SDK block

1 year ago

I've only had one Kinect to play with, but I believe the SDK only supports one device. With the OSC app, you could obviously get data from multiple machines. One of the open source ones supports multiple devices.

Almost finished cleaning up the current OSC app. We've been testing it with Flash and it's crazy fast. With the Away3D 4 API in Flash Player 11, it's very easy to hook the skeleton up to a 3D model. We changed up the address system a bit to make the messages shorter and more obvious.

We're working out a way to transmit data from the depth image. It's far too much data to send XYZ for every point in the image, but we're thinking of treating users more like multi-touch blobs. That is, use OpenCV to give a position (centroid), width, height, ID, and points in the outline (after contour smoothing to reduce the number of points in the outline). Based on doing a lot of work with multi-touch over OSC, I feel this is a reasonable approach for getting useful data without cramming too much data into messages.

Then we'll work out audio. Obviously no need for color data. Toggling the tracking feature (eg, depth, skeletons, etc) will intrinsically determine if message are broadcast. If the depth image is disabled, user outlines won't track. Same for skeletons and audio.

Any other ideas? Anything wrong with that plan?

http://www.BanTheRewind.com

letsgoout..

Re: Kinect SDK block

1 year ago

0.0.8 is up. When aligning skeletons with depth image data, I'd found that four of six user IDs did not match their skeleton IDs. I've updated the block to color the users correctly. I've also added a getUserIdByColor() method with three overloads for Colorf, float, and uint32_t. It's a 1-index to match the SDK's format, where 0 is no user.

This is handy for splitting depth data out to separate users when iterating the depth surface.

http://www.BanTheRewind.com

gabrielst..

Re: Kinect SDK block

1 year ago

Hey !

This is good !

Thanks for the work.

Anyone work on a OSX version ? What would you go for ?

Thanks

letsgoout..

Re: Kinect SDK block

1 year ago

The Kinect SDK is only available for Windows 7, therefore the wrapper is, too. However, as makemachine demonstrated, you can run it fine through Bootcamp.

http://www.BanTheRewind.com

h.mehrpoo..

Re: Kinect SDK block

1 year ago

Hi,
First of all thanks for the Kinect block.
I installed it as you explained on http://www.BanTheRewind.com. it doesn't work properly with Kinect SDK beta2. the microsoft samples works fine. after changing the include files and libs in the project configuration and changing the xcopy in build events it runs.
the Skeleton and audio part works fine. but the depth data and rgb data is not working.

Could you suggest me what to do?

Many thanks,
Hadi

letsgoout..

Re: Kinect SDK block

1 year ago

> the Skeleton and audio part works fine. but the depth data and rgb data is not working.

Sometimes the depth camera won't initialize. Below is are excerpts from a guide I wrote for a Kinect-based app for an installation where someone else had to set it up:

Using a Kinect sensor with a PC is still considered an experimental technology. While it has proven to perform consistently once connected, it has been known to start up inconsistently if it is connected out of sequence. This machine already has Kinect sensor drivers properly installed, so there will never be a need to update them. We’ve had the best luck getting the Kinect sensor to run consistently by following this sequence:

Turn on the computer.
Let all stages of start up finish completely (wait until tray icons stop appearing)
Assuming the Kinect sensor is already connected to the USB/power adapter, plug the USB cable into the computer.
Wait a few seconds.
Plug in the Kinect sensor’s power to an outlet.
Wait until the machine recognizes the sensor before opening any software which uses it. It will not always make it clear that it has recognized it, so just waiting about ten seconds will do the trick.

From the troubleshooting section...

Application starts, but does not detect users

Check that the sensor is powered and plugged into the computer.
Make sure nothing is obstructing the view of the Kinect sensor.
If the depth sensor is not working while the rest of the unit is

Restart the computer.
Test the sensor again.
If it still does not work

Power down the machine.
Unplug the Kinect sensor from the machine and its power.
Power the machine back up and let it complete all startup processes.
Connect the sensor to the machine.
Wait a few seconds.
Connect the sensor’s power.
Wait a few seconds.
Run the test again.
Run the application when ready.

http://www.BanTheRewind.com

h.mehrpoo..

Re: Kinect SDK block

1 year ago

Hi,

Thank you for your reply.
I tried reinstalling the drivers again but still get the same problem.
The Microsoft samples work fine on the machine i.e. it shows depth/rgb and skeleton.
but the block just detects the audio and skeleto. i put breakpoints in the process depth and rgb and it goes there and it shows it already capturing. i was thinking maybe we should change the quad functions.? my USB version is 3? could it be the case?
I'm using Microsoft Beta 2.

Looking forward to hear from you.

Regards,
Hadi

letsgoout..

Re: Kinect SDK block

1 year ago

~~I haven't tested it with the November release yet. That might be the problem. I just downloaded it and will give it a shot on Wednesday.~~

Actually, I'm using a more recent version...

http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/

http://www.BanTheRewind.com

h.mehrpoo..

Re: Kinect SDK block

1 year ago

Thank you,

I can't really go to the url of research.microsoft. it just redirects to the beta SDK. also in the microsoft website there's a topic that shows how to migrate from the research verison to Beta version...

Your code is amazing man. I learned alot more about cinder and boost : )

looking forward to hear from you on wednesday.

letsgoout..

Re: Kinect SDK block

1 year ago

I spent a good chunk of the weekend running through this with a fine-toothed comb. I've been wanted to get a mature, stable release of this block out there for awhile and I'm feeling like it's ready. Check out the first post or follow this link to get it:

http://www.bantherewind.com/kinectsdk-for-cinder

The punched list:

Works with the new Kinect SDK 1.0 beta 2
Support for up to eight connected devices
Camera tilt
Greyscale mode
Improved performance and stability
Added PointCloudGpu sample application
Block arranged to meet new CinderBlock guidelines and improve portability
Includes improved AudioInput class for Windows
Block and samples exit with zero memory remaining
Fixed bug with high resolution modes

I recommend reading the article at bantherewind.com for some info on how to keep the infrared emitter from quitting on you.

http://www.BanTheRewind.com

h.mehrpoo..

Re: Kinect SDK block

1 year ago

Awesome,
Thanks a lot.

will have a look at the article on your website now.

letsgoout..

Re: Kinect SDK block

1 year ago

http://www.bantherewind.com/kinectsdk-for-cinder

Updated to 1.1.0. SDK 1.0 implemented and the block is available as a static library. More resolutions supported more effectively. New SDK / drivers from MS are solid. As long as you don't have anything else on the same hub as your device(s), it runs quickly and consistently.

I've also added a rudimentary "MeshApp" sample. This creates a VBO mesh of triangles and maps color data to it. It's pretty rudimentary, but will be a good starting point. You can also turn off the color in the params to see the UV mapping. I wanted to get this one really polished, but I just haven't had time.

Aiming to overhaul the terribly outdated GitHub repo by the end of the weekend. Will get some screenshots of the mesh app, update the page, etc.

A COUPLE NOTES...

Be sure to restart your computer after installing the new SDK -- before trying to build a sample. For some reason the environment variable for the SDK directory isn't getting picked up by Visual Studio until you restart.

You have to turn down or off the video stream in order to use 640x480 depth. Skeletons can't be tracked at 640x480. This is a limitation of the SDK. At least when using a XBox-specific device. Need to get a Windows one...

http://www.BanTheRewind.com

eclay.jqu..

Re: Kinect SDK block

1 year ago

This is awesome. Thanks for putting in so much hard work on this wrapper.

One quick question I can't seem to figure out is how you linked the Kinect.cpp file from the src directory into the samples. Mainly because I'm trying to modify the wrapper to support Near Mode for the K4W sensor. But I can't figure out how you linked that file into the projects, because it's not actually being compiled as part of the sample (KinectApp for VC++2010 in this case).

Sorry for the n00bish question.

Edit: See, I am a noob. I didn't realize you were linking to the library through a separate project. Anyway, if you want to put this feature into the standard distributions you can modify the OpenDepthStream() from your wrapper to the following:

bool Kinect::openDepthStream()
{
if (mSensor != 0) {
if ( FAILED( mSensor->NuiImageStreamOpen( mDepthResolution != ImageResolution::NUI_IMAGE_RESOLUTION_640x480 && HasSkeletalEngine( mSensor ) ? NUI_IMAGE_TYPE_DEPTH_AND_PLAYER_INDEX : NUI_IMAGE_TYPE_DEPTH, mDepthResolution, 0, 2, 0, & mDepthStreamHandle ) ) ) {
trace( "Unable to open depth image stream" );
stop();
return false;
}
// Set NEAR Mode to enable
mSensor->NuiImageStreamSetImageFrameFlags(mDepthStreamHandle,NUI_IMAGE_STREAM_FLAG_ENABLE_NEAR_MODE | NUI_IMAGE_STREAM_FLAG_DISTINCT_OVERFLOW_DEPTH_VALUES);
}
return true;
}

letsgoout..

Re: Kinect SDK block

1 year ago

Ah, nice. Will add this in as an option.

http://www.BanTheRewind.com

rain2012

回复: Kinect SDK block

1 year ago

he! letsgooutside
I find a error and see

PointCloudApp in MeshApp

letsgoout..

回复: Kinect SDK block

1 year ago

Picture isn't showing up. Not sure what you mean.

http://www.BanTheRewind.com

joshuajno..

Re: Kinect SDK block

1 year ago

Nice work! Makes me glad I've got a Win7 OS to boot into now :) The uptime issue around the USB hubs issue is pretty bizarre, since if I remember right, the FreeNect stuff doesn't do that, so it's a bit bizarre. I'd love to know if someone has better luck w/the Kinect for Win machines.

http://thefactoryfactory.com

letsgoout..

Re: Kinect SDK block

1 year ago

I've started overhauling my GitHub account to make it a little easier to navigate and more open to collaboration. The KinectSdk one is now here:

https://github.com/BanTheRewind/Cinder-KinectSdk

This will be the place to report bugs, add features, etc, rather than the Cinder forum.

http://www.BanTheRewind.com

letsgoout..

Re: Kinect SDK block

1 year ago

Just posted an update to the block. I've added "near mode" per eclay.jquery's suggestion. The KinectApp sample shows how to use it. Note that this will only work on Kinect for Windows devices. It does nothing on the XBox device.

I've also improved threading a bit. I've replaced scoped_locks with lighter weight lock_guards, brought down the WAIT_TIME for info to 175ms from 250ms, and added a sleep routine to bring down CPU usage (pointed out by sharkbox).

http://www.BanTheRewind.com

coder.guy

Re: Kinect SDK block

1 year ago

Hello All,

I just updated to the latest version of this block and am having inconsistent results with all of the samples. The primary problem seems to be that the kinect thread keeps dying.

"The thread 'Win32 Thread' (0x1374) has exited with code 0 (0x00)."
"The thread 'Win32 Thread' (0x6bc) has exited with code 0 (0x00)."
"The thread 'Win32 Thread' (0xf08) has exited with code 0 (0x00)."

etc...and then the app locks up after a thread dies with code 1.

I'm running stock cinder 83 with the latest Kinect drivers installed (v.1.0). Using an Xbox 360 Kinect that runs fine with the SDK Skeleton viewer example and runs better with the previous version of the block although at ~70% CPU on most of the samples there but i'm more concerned with the consistency of the samples in the latest block. Any ideas on how to try to get better performance or consistency? I'm running win7 on a mac mini with 8GB or RAM.

Thanks,

CG

letsgoout..

Re: Kinect SDK block

1 year ago

The block hasn't been officially tested under Boot Camp, so there might be issues there. I made some updates to reduce CPU consumption which worked well on my PC, but might have adverse effects on a Mac running Windows. First thing I'd recommend is removing all other USB devices while the Kinect is plugged in.

http://www.BanTheRewind.com

coder.guy

Re: Kinect SDK block

1 year ago

Thanks for the reply...i'll find a PC to run some tests... pulling the USB devices out had no impact....

The Standard Kinect VS Express SkeletonTracker app runs at about 15% CPU in boot camp so i figured there was something odd....

-CG

letsgoout..

Re: Kinect SDK block

1 year ago

What is your CPU? The thing I added recently might be more platform-specific than I first realized.

http://www.BanTheRewind.com

coder.guy

Re: Kinect SDK block

1 year ago

Greetings,

I've tested on two systems with similar results. The mac mini i was testing uses a 2.7GHz Dual-Core Intel Core i7 w/8GB of RAM. I also tried it on a windows box with lower specs but found similar results...the windows box had an intel core i3-540@3.06GHz and 4GB of RAM. Both ran the basic Kinect app at about 70-80% CPU, sometimes hitting 100% (using only a single Kinect). I also tried to build using master as was suggested by Paul Houx as i thought the issue might be the openCV example i was trying to build from (Andy Berg's hand tracking example)...but the block examples alone also spike CPU utilization....

What type of CPU utilization do you see on your system for the basic example in the block? What are your test system specs? It is odd because i even tried to only process kinect data every 20th frame (app running at 30fps) and saw no drop off in CPU utilization, perhaps it is a threading issue? (purely naive speculation on my part, fyi but trying to help)...I could actually deal with the CPU utilization but i've also had issues using 2 Kinects. When trying to only access the 2nd Kinect the app crashes pretty hard. The only thing i changed was the device index in the sample app. The windows skeleton viewer app works fine with two kinects, toggling between both and runs at about 15% CPU.

I understand that the USB controllers play a role but i verified on the Windows PC that both Kinects were on their own USB host controllers so i'm at a bit of a loss.

any information you could provide would be greatly appreciated.

kindest regards,

CG

letsgoout..

Re: Kinect SDK block

1 year ago

Thanks for the info. This will help a lot. I'm going to dig into this hard over the weekend. I've got three very different machines set up to test against, including a Mac with Boot Camp. Read up on potential issues with threading and have some theories to try out. The second Kinect crashing and your high CPU utilization are likely linked.

http://www.BanTheRewind.com

coder.guy

Re: Kinect SDK block

1 year ago

Thanks a ton. Let me know if there is any way I can help.

-CG

letsgoout..

Re: Kinect SDK block

1 year ago

Just did a push:

https://github.com/BanTheRewind/Cinder-KinectSdk/commit/4dcbcb8dcae9d356e5060c32cb97d0121bae9c9e

I was able to get the symptoms you mentioned on an older machine. I moved all the processes into a single thread and wrote a boost-y-er way to pace the thread so it's consistent across platforms. Too many concurrent threads can have adverse effects. While the previous versions worked great on my i7 machines, it wasn't fairing so well on lesser chips.

I noticed the spikes. It turns out this started happening in the Beta 2 SDK and persists in the 1.0 version. There are a lot of complaints about this on the web and no obvious solution. That code is buried in the DLL.

What you should be seeing now is probably closer to 15% - 20% on average for the SDK app with everything turned on (add 5% - 10% when tracking a busy image). When you first start any application, the SDK's init routines will run up an additional 30% - 40% for a moment. As you watch the CPU usage, you'll the SDK's occasional spikes. They're very brief and shouldn't run up too high. Hopefully they'll be fixed in the next release.

I've also noticed that the skeleton tracker has used more and more CPU with each release, it seems. If you don't need skeletons, I highly recommend turning them off to get your frame rate up.

Last thing to note is that the changes I posted introduced an issue where once you stop a device, you can't restart it. Everything else seems to toggle fine. I'll fix this soon but need to sign off for tonight. Just wanted to get something up before the work week starts. Let me know if you find anymore issues.

http://www.BanTheRewind.com

safetydank

Re: Kinect SDK block

1 year ago

I had a look at your diff and I'm unsure about your mutex locking. When you construct a lock_guard like:

if (mRunning) {
boost::lock_guard<boost::mutex> lock(mMutex);
}

The lock is released at the end of the if statement, not at the end of the function as I think you intend. I checked the scoping with GCC, maybe it works with MSVC?

letsgoout..

Re: Kinect SDK block

1 year ago

Work in progress. Those are there ahead of fixing the bug where you can't restart.

http://www.BanTheRewind.com

acchapman

Re: Kinect SDK block

1 year ago

This is fantastic. A nice improvement on what was already great.

I've been using it for some 3d projects, but now want to implement some simple 2D examples where the skeleton is directly overlaid onto the video (instead of drawing in 3space). Is there an easy way to do this that I'm missing?

Also, what's the difference between MSR_NuiApi and NuiApi? Am I using an outdated library?

letsgoout..

Re: Kinect SDK block

1 year ago

You can use the camera to project world space back to screen coordinates. It's a bit slow, though, so you might want to just do one joint and use the difference to scale the rest without the worldToScreen conversion.

Yes, MSR_NuiApi is obsolete. I think there was even another one before that, if memory serves me correctly.

Aiming to wrap up the restart bug by next weekend. Have some higher priority items before then.

http://www.BanTheRewind.com

acchapman

Re: Kinect SDK block

1 year ago

thanks. that's super helpful.

So I updated all the drivers and installed the new BTR block. Everything works great, except when it doesn't.

I'm running Win7 on a MacBook Pro (and that the old versions worked without any problems).

I think there may be two issues:

1) the new MS drivers seem to periodically crash/lag in skeleton mode. not sure there's anything to be done about that.

2) I think you have some threading issues.

Starting up your KinectApp:

Often the Kinect fails to start, or lags significantly before it starts.

Whenever a Skeleton is identified, it crashes: usually hanging on a un/lock. When I have Skeleton tracking turned off, everything runs incredibly smoothly (faster framerate, less processor hit than the last version). Depth is fine, Video is fine, Audio is fine.

I hope this helps. What you're doing here is wonderful.

letsgoout..

Re: Kinect SDK block

1 year ago

Thanks for the detailed report. Will have time this coming weekend to get Kinect on Boot Camp up to speed.

http://www.BanTheRewind.com

letsgoout..

Re: Kinect SDK block

1 year ago

Try pulling from the repo now. Made some huge improvements in performance and CPU usage. I was actually able to remove the use of a mutex completely. This resolves a long running issue where requesting the user count would cause your app frame rate to match the depth frame rate. The two threads run completely independent of each other.

You should be seeing CPU usage in from low twenties to the low thirties (depends, of course) when tracking a busy scene. More than half of the CPU usage is *just* the SDK. In the KinectApp sample, add 5% - 10% for audio. You'll see CPU drop when you turn it off.

Unfortunately the restart bug is still there. Stabilizing the performance actually took me further away from resolving that one. :/ Still working at it.

http://www.BanTheRewind.com

acchapman

Re: Kinect SDK block

1 year ago

This is great. much smoother.

So, I'm still being dense about doing something that I think is simple. I just want to do a direct overlay of the skeleton onto the video at roughly 1:1. worldToScreen (unless I'm wrong) simply translates from 3space to screen 2space.

What I need to do is more like this, where the skeleton is mapped directly onto the depth image and video. -- I got the depth image to overlay the video, pretty much exactly, but the skeleton overlay is always offset pretty significantly - especially at the edges.

Is there a simple way to do this?

letsgoout..

Re: Kinect SDK block

1 year ago

It depends on how you've set up your camera. I'm away from a Kinect right now, but something like this should do it...

In setup ( mCamera is a ci::CameraPersp )...

mCamera.lookAt( Vec3f( 0.0f, 0.0f, -2.0f ), Vec3f::zero() );
mCamera.setPerspective( 45.0f, getWindowAspectRatio(), 0.001f, 100.0f );

In draw...

gl::clear();
gl::setViewport( getWindowBounds() );
// Draw depth texture (2D)
gl::pushMatrices();
gl::setMatricesWindow( getWindowSize() );
gl::draw( mDepthTexture, getWindowBounds() );
gl::popMatrices();
// Draw skeleton (3D)
gl::pushMatrices();
gl::setMatrices( mCamera );
// Draw skeleton here, as in example
gl::pushMatrices();

Just wrote the code here in the forum, so double check it. Do not scale the skeleton positions. The key here will be to find an eye position ( first argument in CameraPersp::lookAt() ) that lines the skeleton up with the 2D image. I believe this will do it. If you don't see the skeleton at all, try a positive Z value.

That should at least get the size right. You may need to add an overall offset to the skeleton to line it up perfectly. The above code would be modified like so:

// Draw skeleton (3D)
gl::pushMatrices();
gl::setMatrices( mCamera );
gl::translate( mSkeletonOffset );
// Draw skeleton here, as in example
gl::pushMatrices();

Good luck!

http://www.BanTheRewind.com

acchapman

Re: Kinect SDK block

1 year ago

This isn't exactly 1:1, but it's pretty close and seems to work if I place the camera in many places at many positions. That said, I wouldn't be surprised if there's a more efficient way to do this.

mCamera.lookAt( Vec3f( 0.0f, 0.0f, -1.0f ), Vec3f::zero() );
mCamera.setPerspective( 1.0f, getWindowAspectRatio(), 1.0f, 1000.0f );

in draw:

gl::setMatrices( mCamera );
gl::pushMatrices();
gl::scale( mKinectSkeletonScale, mKinectSkeletonScale, mKinectSkeletonDepthScale );
// for every joint
// gl::vertex( skeleton.at( joints[ i ] ) * Vec3f( -1.0f, 1.0f, 1.0f ) );
drawSkeletons3d();
gl::popMatrices();
//2d
mScreenW = getWindowWidth();
mScreenH = getWindowHeight();
gl::setMatricesWindow( getWindowSize(), true );
gl::enableAlphaBlending();
glPushMatrix();
// DRAW DEPTH -----------------------------------------------------
if( mDepthSurface){
// MUST shift image because depth and video are naturally offset
gl::translate( kinectNaturalDepthOffsetX, kinectNaturalDepthOffsetY, 0.0f);
gl::scale( kinectNaturalDepthScale, kinectNaturalDepthScale, kinectNaturalDepthScale );
gl::color( ColorA( 1.f, 0.5f, 0.0f, 0.75f ) );
gl::draw(gl::Texture(mDepthSurface), Area(0, 0, mDepthSurface.getWidth(), mDepthSurface.getHeight()), Rectf( 0.f, 0.f, mScreenW, mScreenH ) );
}
// for every joint
// gl::vertex( mCamera.worldToScreen( (skeleton.at( joints[i] ) * Vec3f( -0.03f, 0.03f, 1.0f) ), mScreenW, mScreenH ) );
drawSkeletons2d();
glPopMatrix();

letsgoout..

Re: Kinect SDK block

1 year ago

I think rather than worldToScreeninging every joint, you could simply use two vectors -- scale and offset -- for both the depth image and skeleton. You're already doing that for the depth image. An adjustable offset and scale will give you a way to manually calibrate your image, thus not incurring the penalty that comes with worldToScreen.

mCamera.setPerspective( 1.0f, getWindowAspectRatio(), 1.0f, 1000.0f );

That first parameter is the camera's field of view angle. Is 1° working?

Good luck! Glad to hear it's coming along.

http://www.BanTheRewind.com

acchapman

Re: Kinect SDK block

1 year ago

yeah, 1° seems to provide the least distortion toward the edges. Of course, I have to scale everything way down to view it: 0.03 on x and y. I know it seems weird, but the lower the FOV, the lower the distortion/offset toward the edges. With a normal 45° - 90° FOV the skeleton starts to offset on the edges a lot. I'm trying to test for different camera locations/setups, angles, and room sizes/stages. -- But again, I'm probably doing something wrong.

It makes sense to do just the basic offset. thanks. Way better than all repeated matrix multiplies.

daalruag

Re: Kinect SDK block

1 year ago

I saw a thread a possible solution for the error of Cinder-block xcopy KinectSdk, I saw that is a matter of updating the sdk dir but not working properly, change the path before me said she could not include "NuiApi.h" to modify the path and decision to run the program but throws me errors mInput

error C2065: 'mInput': undeclared identifier

Error C3861: 'create': identifier not found.

I'm using the last-KinectSdk Cinder block and the latest version of the sdk v1.0 kinect, the examples that comes with the SDK work well but the cinder block still does not work.

Thanks in advance

letsgoout..

Re: Kinect SDK block

1 year ago

I saw a thread a possible solution for the error of Cinder-block xcopy KinectSdk, I saw that is a matter ofupdating the sdk dir but not working properly, change the path before me said she could not include "NuiApi.h"to modify the path and decision to run the program but throws me errors mInput

Be sure to restart your computer after installing the Kinect SDK from Microsoft. The path variables are not set until you restart.

error C2065: 'mInput': undeclared identifier
Error C3861: 'create': identifier not found.

Is mInput your Kinect device or the AudioInput in the sample?

If it is audio, be sure to grab the Windows audio input block here:

https://github.com/BanTheRewind/Cinder-AudioInputWin

If mInput is your Kinect device, note that there is a KinectSdk namespace. You need to either...

Add this line to your implementation:
using namespace KinectSdk;
Prefix your Kinect objects with the namespace:
KinectSdk::KinectRef mInput;
...
mInput = KinectSdk::Kinect::create();

Good luck!

http://www.BanTheRewind.com

daalruag

Re: Kinect SDK block

1 year ago

@ letsgooutside

thanks for your help I managed to correct los problemas anteriores, I build solution and all is good but when I run the program works well but does not show the depht or rgb, i guess i´ts of version of sdk, in the output sohws me that error:

c:\code\cinder_0.8.3_vc2010\src\anttweakbar\twopengl.cpp(240) : [CTwGraphOpenGL::BeginDraw] GL_ERROR=0x500

when try to plot of two cameras. maybe give you idea on the problem.

letsgoout..

Re: Kinect SDK block

1 year ago

It sounds like you are trying to draw the params after calling gl::pushMatrices() without a matching gl::popMatrices().

Wrong:

gl::pushMatrices();
gl::translate( 1.0f, 1.0f, 0.0f );
gl::draw( mTexture );
params::InterfaceGl::draw(); // <-- error

Right:

gl::pushMatrices();
gl::translate( 1.0f, 1.0f, 0.0f );
gl::draw( mTexture );
gl::popMatrices();
params::InterfaceGl::draw();

http://www.BanTheRewind.com

daalruag

Re: Kinect SDK block

1 year ago

I tried modified some things but I get the same problem, in fact the example of kinectApp throws me the same error.

letsgoout..

Re: Kinect SDK block

1 year ago

c:\code\cinder_0.8.3_vc2010\src\anttweakbar\twopengl.cpp(240) : [CTwGraphOpenGL::BeginDraw] GL_ERROR=0x500

The error you are having isn't specific to this block, but with the parameters. Try commenting out the params and see how it runs. Have you added any new parameters to params? Did you initialize all the values you are editing? Did you make sure you called addParam, addButton, and/or addText correctly (ie, don't call addText if you mean to add a parameter).

I recommend starting a new thread for this issue so it doesn't make this one longer than it needs to be. ;)

http://www.BanTheRewind.com

daalruag

Re: Kinect SDK block

1 year ago

ok thanks new thread ;)

daalruag

Re: Kinect SDK block

1 year ago

@ letsgooutside.

well I finally made the errors are gone working on x64 x32 I think that does not work wellI'll be looking AntTweakBar to prove my hypothesis. :) here is the proof that is already running.

xumerio

Re: Kinect SDK block

1 year ago

I would appreciate some help with this:

I have two Kinects conected but cannot get both depth images at the same time.

Only one is working well and I get this message in the console:

The thread 'Win32 Thread' (0x744) has exited with code 0 (0x0).

The thread 'Win32 Thread' (0xe00) has exited with code 0 (0x0).

The thread 'Win32 Thread' (0xc0c) has exited with code 0 (0x0).

The thread 'Win32 Thread' (0x94c) has exited with code 0 (0x0).

The thread 'Win32 Thread' (0x788) has exited with code 0 (0x0).

The thread 'Win32 Thread' (0x824) has exited with code 0 (0x0).

This is more or lees the code:

declare both:

/*Para Kinect*/
KinectRef kinectA;
KinectRef kinectB;
gl::Texture depthTextureA;
gl::Texture depthTextureB;

setup():

cuantos = Kinect::getDeviceCount();
console() << "There are " << cuantos << " Kinects connected." << std::endl;
kinectA = Kinect::create();
kinectA->start(0,ImageResolution::NUI_IMAGE_RESOLUTION_640x480);
kinectA->enableDepth(true);
kinectA->enableVideo(false);
kinectA->enableBinaryMode(false,false);
if(cuantos>1)
{
kinectB = Kinect::create();
kinectB->start(1,ImageResolution::NUI_IMAGE_RESOLUTION_640x480);
kinectB->enableDepth(true);
kinectB->enableVideo(false);
kinectB->enableBinaryMode(false,false);
}

update:

if( kinectA->checkNewDepthFrame() )
{
depthTextureA = kinectA->getDepth();
}
if( kinectB->checkNewDepthFrame() )
{
depthTextureB = kinectB->getDepth();
}

draw:

if(depthTextureA)
{
gl::draw(depthTextureA);
}
if(depthTextureB)
{
gl::draw(depthTextureB);
}

btw, this is my first project in cinder and it is awesome.

letsgoout..

Re: Kinect SDK block

1 year ago

I finally have time to address this this week. This is a known issue which appeared after making some updates to reduce CPU usage. Apologies.

http://www.BanTheRewind.com

todd_9

Re: Kinect SDK block

1 year ago

Hi, first let me say thanks for all of this so far.

I'm having trouble compiling any of the samples. Any help would be much appreciated. Here is my output:

1>------ Build started: Project: KinectApp, Configuration: Release Win32 ------
1> C:\Program Files\Microsoft SDKs\Kinect\v1.0\\Assemblies\Microsoft.Kinect.dll
1> 1 File(s) copied
1> KinectApp.cpp
1>C:\Users\Todd\Documents\Visual Studio 2010\cinder_master\blocks\Cinder-KinectSdk\src\Kinect.h(80): warning C4482: nonstandard extension used: enum '_NUI_IMAGE_RESOLUTION' used in qualified name
1>C:\Users\Todd\Documents\Visual Studio 2010\cinder_master\blocks\Cinder-KinectSdk\src\Kinect.h(81): warning C4482: nonstandard extension used: enum '_NUI_IMAGE_RESOLUTION' used in qualified name
1>C:\Users\Todd\Documents\Visual Studio 2010\cinder_master\blocks\Cinder-KinectSdk\src\Kinect.h(111): warning C4482: nonstandard extension used: enum '_NUI_IMAGE_RESOLUTION' used in qualified name
1>C:\Users\Todd\Documents\Visual Studio 2010\cinder_master\blocks\Cinder-KinectSdk\src\Kinect.h(112): warning C4482: nonstandard extension used: enum '_NUI_IMAGE_RESOLUTION' used in qualified name
1>C:\Users\Todd\Documents\Visual Studio 2010\cinder_master\blocks\Cinder-KinectSdk\src\Kinect.h(80): warning C4482: nonstandard extension used: enum '_NUI_IMAGE_RESOLUTION' used in qualified name
1>C:\Users\Todd\Documents\Visual Studio 2010\cinder_master\blocks\Cinder-KinectSdk\src\Kinect.h(81): warning C4482: nonstandard extension used: enum '_NUI_IMAGE_RESOLUTION' used in qualified name

...

letsgoout..

Re: Kinect SDK block

1 year ago

It looks you are using the version of Cinder on Github, which uses fs::path instead of strings for file paths. This block is designed to work with Cinder 0.8.3 at the moment. I'm actually updating the block right now to work with Cinder 0.8.4.

In the mean time, change the "screenShot" method in the samples to the following:

void PointCloudApp::screenShot()
{
writeImage( getAppPath() / fs::path( "frame" + toString( getElapsedFrames() ) + ".png" ), copyWindowSurface() );
}

Please note that you will also need build the Kinect library against the latest Cinder so it uses boost 1.48.

http://www.BanTheRewind.com

todd_9

Re: Kinect SDK block

1 year ago

Eureka! That was it. Thank you.

letsgoout..

Re: Kinect SDK block

1 year ago

Good to hear. Have fun with it!

http://www.BanTheRewind.com

letsgoout..

Re: Kinect SDK block

1 year ago

Block has been updated to work with Cinder 0.8.4. This also has been tested against Kinect SDK 1.0.3.191 which was released yesterday.

https://github.com/BanTheRewind/Cinder-KinectSdk

There was a potential redistribution error in The Kinect SDK 1.0.2 which the new, minor release addresses.

http://www.BanTheRewind.com

todd_9

Re: Kinect SDK block

1 year ago

Just wanted to let you know that your latest works fine for me with the May 1st Microsoft Kinect SDK and the latest Cinder pull from Github. Thanks for the update!

Edit: The "KinectApp" sample won't compile for me, but all the others do.

One other thing... I haven't fully investigated it yet, but it appears only a fraction of the total available depth precision is being used in the examples. Depth values are currently 8-bit, but they should be 12-bit.

todd_9

Re: Kinect SDK block

1 year ago

So, it looks like Kinect::shortToPixel() in Kinect.cpp is throwing away a lot of the depth precision. Here's an experiment comparing the "intensity" value currently being used (above) with the "realDepth" value (below):

I modified the function to put the original 8-bit value in the BLUE channel, and the raw 12-bit value in the R and G channels (which is very hacky but at least I was able to confirm my suspicions).

letsgoout..

Re: Kinect SDK block

1 year ago

I've been meaning to switch the depth image from Surface8u to Surface16u, but it's slipped my mind. I've been more focused on CPU, OSC, etc. It's an easy change so I'll do it today.

http://www.BanTheRewind.com

sixswitch

Re: Kinect SDK block

1 year ago

This would be awesome!

xumerio

Re: Kinect SDK block

1 year ago

Great news @letsgooutside .Wil try this as soon as I get back the Kineck (We are doing tests and setting the instalation).

todd_9

Re: Kinect SDK block

1 year ago

Looking forward to those extra bits!! Nice work with this block... very appreciated. Here's a little video of what I've managed to hack together so far:

https://vimeo.com/41633902

daalruag

Re: Kinect SDK block

1 year ago

hello when you download the latest package from the GitHub cinder is not the folderboost and some applications annoying;). as kinectApp

https://github.com/cinder/Cinder

sixswitch

Re: Kinect SDK block

1 year ago

In case it helps anyone else out: I'm running this via Boot Camp, and my Apple keyboard caused a conflict because it doubles as a USB hub. Swapping out for a plain USB keyboard got everything up and running!

letsgoout..

Re: Kinect SDK block

1 year ago

Yes. I've brought that up on here and on the blog, but it's worth pointing out again. Kinects do not like to share when it comes to USB. Hard drives, wireless mice, etc, will often keep your device from starting up when plugged into the same hub. I generally only run Kinects and nothing else when working with them.

http://www.BanTheRewind.com

timmb

Re: Kinect SDK block

1 year ago

Hi Stephen

I was having a few difficult to identify exceptions arising with my Cinder program using you Kinect SDK and think I've tracked it down to a race condition when the mSkeletons vector was being written to by your thread at the same time as when my application was copying it using the getSkeletons() method.

A simple solution that has solved the problem for me was to ensure that the mNewSkeletons variable is not set until false until after mSkeletons has been copied like this:

void Kinect::getSkeletons(std::vector<Skeleton>* out)
{
*out = mSkeletons;
mNewSkeletons = false;
}

rather than returning a reference.

Tim

letsgoout..

Re: Kinect SDK block

1 year ago

The SDK block is undergoing a major update right now. I'll have a fork for people to play with in the next day or two.

http://www.BanTheRewind.com

eight_io

Re: Kinect SDK block

1 year ago

Are you considering sending out the user pixels via some protocol? I found that current OSC implementation does not seem to be adequate for this task, without severe subsampling. Trying to send the user pixels using asio along these lines, which still may require subsampling it seems. A good news is that, sending the frame difference instead of the entire frame, makes the amount of data to drop 10-fold in my experiments (in this case OSC is still inadequate without subsampling though, even with zlib compression).

--8

Analysis: http://post.scriptum.ru
Synthesis: http://onewaytheater.us

letsgoout..

Re: Kinect SDK block

1 year ago

For OSC, I'm just thinking skeletons and outlines (as vectors). You'll be able to specify the outline simplification to increase performance.

http://www.BanTheRewind.com

zuppaman

Re: Kinect SDK block

10 months ago

Hey,

@letsgoout really fantastic work!

This plugin and samples have been really helpful.

They are much more readable then the MS samples.

Everything compiles and runs fine except the multiple device sample.

It starts but I got no frames from the second device although it gets recognized.

At first I thought it was a hardware issue but then I modified the depth tracking wpf sample from MS.

That worked fine with 2 kinects so there must be something wrong in the cinderblock sample...

The skeleton tracking seems to be a problem on all samples (WPF,C++ from MS & Cinder bock) if you try to use two devices at once.

After some googling it seems to be a problem if you run skeleton tracking in one thread.

Running the skeleton samples (c#,C++, cinder) seperatly preconfigered for a specific Kinect instance works fine.

So conclusion it's not a show stopper for me because I was already planning to seperate things with osc so running multiple instances is not a problem.

Greets

Kris

letsgoout..

Re: Kinect SDK block

10 months ago

There was a small change made after the last time I tested with multiple devices that introduced this. Not only can skeletons not be tracked on multiple devices, neither can users. That is, with multiple devices, you can't do player indexing. I've introduced enableUserTracking and isUserTrackingEnabled methods to DeviceOptions. Just pushed to master branch. This is separate from skeleton tracking, but skeleton tracking does rely on it. Disable user tracking when using multiple device and you'll be good to go.

http://www.BanTheRewind.com

letsgoout..

Re: Kinect SDK block

10 months ago

I should amend my statement to say that you can't track users on additional devices. I was able to track users with multiple devices, as long as I did so on only the first connected device (index = 0). Also note that user tracking is only supported for 320x240 or 80x60 modes. If you set your depth to 640x480, skeleton and user tracking will automatically be turned off. This behavior was already in there (and is mentioned in the comments in the header), but thought I'd bring it up here, as well.

http://www.BanTheRewind.com

zuppaman

Re: Kinect SDK block

10 months ago

Wow thx for this super fast fix.

Runs lighting fast!!

joshuajno..

Re: Kinect SDK block

9 months ago

I just tried installing this on a new machine and ran into the error in the image below (can't copy/paste text from VS errors). This block used to work fine on my old machine, so I'm wondering if this is somehow related to my installing the 1.5.1 SDK?

http://thefactoryfactory.com

letsgoout..

Re: Kinect SDK block

9 months ago

It looks like you are trying to run the library as an application. If you are using the "all" solution, be sure to set one of the samples as your startup project.

http://www.BanTheRewind.com

joshuajno..

Re: Kinect SDK block

9 months ago

Yeah, realized that right after posting here, my utter unfamiliarity with Visual Studio is showing I guess :) Thanks for the nice library!

http://thefactoryfactory.com

letsgoout..

Re: Kinect SDK block

9 months ago

Yeah, I have a tendency to do that myself. Especially when toggling between a lib and bin project.

acchapman

Re: Kinect SDK block

9 months ago

This is fantastic. It continues to get better and better.

offhand, does anyone know why when I do this, the video freezes and I get an "out of memory bounds"

for ( vector<Skeleton>::const_iterator skeletonIt = mSkeletons.cbegin(); skeletonIt != mSkeletons.cend(); ++skeletonIt, i++ ) {
Vec3f rightHandPos = skeletonIt->at( NUI_SKELETON_POSITION_HAND_RIGHT ).getPosition();
}

but when I do this, it's fine:

for ( vector<Skeleton>::const_iterator skeletonIt = mSkeletons.cbegin(); skeletonIt != mSkeletons.cend(); ++skeletonIt, i++ ) {
      for ( Skeleton::const_iterator boneIt = skeletonIt->cbegin(); boneIt != skeletonIt->cend(); ++boneIt ) {
            Vec3f rightHandPos = skeletonIt->at( NUI_SKELETON_POSITION_HAND_RIGHT ).getPosition();
      }
}

I'm not using the boneIt, so this confuse me.

letsgoout..

Re: Kinect SDK block

9 months ago

The Kinect SDK tracks up to six simultaneous users, but only provides skeleton data for two, leaving the rest empty. This helps to align skeleton data with player index (eg, mSkeletons[ 3 ] is the skeleton for player #3). The block mirrors this behavior by always returning six skeletons, even if some are empty.

This might see like strange behavior, but it comes in handy in a number of scenarios. For example, if player #2 had a skeleton with bones in the last frame, but has an empty skeleton this frame, you can infer that they have left.

The first loop you posted assumes all skeletons have bones, but that can never be the case. Because it's a map, I believe you can do this:

if ( ( *skeletonIt )[ JointName::NUI_SKELETON_POSITION_HAND_RIGHT ) != skeletonIt->end() ) {
// Do stuff
}

Double check that, but it's something like that.

http://www.BanTheRewind.com

acchapman

Re: Kinect SDK block

9 months ago

Ah, ok. this makes sense. thanks.

This is in place for tonight and doesn't seem to suffer the same problem, but will test more thoroughly tomorrow.

for ( vector<Skeleton>::const_iterator skeletonIt = mSkeletons.cbegin(); skeletonIt != mSkeletons.cend(); ++skeletonIt, i++ ) {
if ( skeletonIt->size() > 0 ) {
//do my thing
}
}

christoph..

Re: Kinect SDK block

8 months ago

When running the KinectApp sample i can see a sudden cut of in depth values at about 4 meters. when looking at kinect sdk samples i can see values that a further away from the kinect sensor, noisy but there are values in the depth image. is there a switch to turn off the depth clamping or is it a cut of in value range when rendering the texture?

letsgoout..

Re: Kinect SDK block

8 months ago

Check if near mode is enabled on one and not the other. Near mode lets you track people very close to the camera, but reduces over all distance.

http://www.BanTheRewind.com

christoph..

Re: Kinect SDK block

8 months ago

In KinectApp enabling Nearmode reduces the maximum depth even more. So its not near mode. Any other ideas?

rvandelden

Re: Kinect SDK block

8 months ago

Seems like great work, haven't tried it myself yet. But did help in solving some issues thanks for your active replies!

I see you have trouble with the six persons and only two skeleton data. A college from my "solved" this by limiting the fps but looping through all the users. Might give it a look at:

https://github.com/Robnocop , Looping-skeletalviewer.

I also see you only seem to use grey values. Might be helpful to include false colors (worked on it yesterday for another platform QT based GUI on top of OpenCV, is fairly simple assumes bitshifted depth image: github parlevision, kinectsdk1 branch, parlevision / src / plvmskinect) although it could slightly slow down your application. I am not a hardcore coder and quite new to it so assume bugs etccetra.

BTW in new SDK you can add up to four Kinects working simultaneously. To limit unplugging your usb stuff: check control panel, device manager, VIEW-> devices by connection. You can see what usb devices run under which port. Helped me a lot especially circumvent using (buidld in) webcam's port might help.

rvandelden

Re: Kinect SDK block

8 months ago

Seems like great work, haven't tried it myself yet. But did help in solving some issues thanks for your active replies!

I see you have trouble with the six persons and only two skeleton data. A college from my "solved" this by limiting the fps but looping through all the users. Might give it a look at:

letsgoout..

Re: Kinect SDK block

8 months ago

I thought I posted a response, but it seems to have disappeared?

A college from my "solved" this by limiting the fps but looping through all the users

So this actually picks out six skeletons on Kinect? Nice. Will definitely check this out.

I also see you only seem to use grey values. Might be helpful to include false colors

User colors are enabled by default. The red channel represents the raw depth data, which blue and green are used to visually differentiate users. If the users are appearing in greyscale to you, try this:

mKinect->enableUserColor( true );

BTW in new SDK you can add up to four Kinects working simultaneously

Actually, it's eight. :) Each device takes up 61% of a USB 2.0 controller, so you need a separate controller for each device. A fast machine with enough PCI slots can run eight devices like a champ.

http://www.BanTheRewind.com

rvandelden

Re: Kinect SDK block

8 months ago

Seems like great work, haven't tried it myself yet. But did help in solving some issues thanks for your active replies!

I see you have trouble with the six persons and only two skeleton data. A college from my "solved" this by limiting the fps but looping through all the users. Might give it a look at:

christoph..

Re: Kinect SDK block

8 months ago

one more problem. when running the KinectApp project from within VS2010 and trying to connect to the Kinect Camera from the Kinect Studio app that comes with the SDK, to record an play the streams, the VS app crashes. When running the released VS app outside of VS, injecting streams with Kinect Studio works fine.

Anyone succeded in running the Kinect Studio together with VS?

It would be awesome if the VS project would not crash, to debug stuff.

paul.houx

Re: Kinect SDK block

8 months ago

Could this be an issue with the working directory? Try setting it to "..":

christoph..

Re: Kinect SDK block

8 months ago

changed the working directory. still crashes whenn hitting connect in Kinect Studio

letsgoout..

Re: Kinect SDK block

8 months ago

I haven't actually tried Kinect Studio yet. I think Paul is onto something. But maybe try setting the working directory to "$(ProjectDir)bin\".

http://www.BanTheRewind.com

christoph..

Re: Kinect SDK block

8 months ago

"$(ProjectDir)bin\" crashes too.

By the way do you have another idea why the depth values are cut off ( in reference to my previous post link) ?

timgroene..

Reageren: Kinect SDK block

6 months ago

I'm in a situation where I need to align the video texture with the depth texture.

I assumed this was automatically done by the Kinect SDK but I was wrong. There's however a method for this called NuiImageGetColorPixelCoordinatesFromDepthPixel which I added to the wrapper.

ci::Vec2i Kinect::getColorPixelCoordinateFromDepthPixel( ci::Vec2i & input ){
long x,y;
mSensor->NuiImageGetColorPixelCoordinatesFromDepthPixel( mDeviceOptions.getVideoResolution(), NULL, input.x, input.y, mDepthSurface.getPixel( input ).r,&x, &y);
return Vec2i( x, y );
}

Calling this for every pixel every time there's a new video texture however destroys my performance. Some optimization can be done by only calling the method for evey depth pixel that corresponds to a user.

This feels like something that can be done using a shader, using the depth texture as a displacement map for the video texture. To be continued..

letsgoout..

Re: Kinect SDK block

6 months ago

Calling this for every pixel every time there's a new video texture however destroys my performance. Some optimization can be done by only calling the method for evey depth pixel that corresponds to a user.

This feels like something that can be done using a shader, using the depth texture as a displacement map for the video texture.

You are correct, sir. I'm aware of this method but found it to be unusably slow. To accomplish this task, use the depth texture to displace a mesh or point cloud of the same pixel dimension (see PointCloudGpu sample). Set the texture coordinates for each vertex so that they go from [0,0] in the top left to [1,1] in the bottom right. Use the video texture to color each vertex.

Now the problem here is that the cameras are not perfectly aligned. Add two vec2 uniforms to your fragment shader which you will use to translate and scale the video texture to line up with your mesh or point cloud.

uniform vec2 scale;
uniform vec2 translate;
uniform sampler2D tex;
varying vec2 uv;
void main( void )
{
gl_FragColor = texture2D( tex, ( uv + translate ) * scale );
}

Set up some params to control the translate and scale uniforms to fine tune the alignment. It's not unlike moving a projector around to line up with a screen.

I recommend setting the video texture to wrap and allow your parameters to go from [-1,-1] to [1,1].

http://www.BanTheRewind.com

timgroene..

Re: Kinect SDK block

6 months ago

Is it really as simple as setting an offset in scale and translation to the fragment shader ? I will certainly try your suggestion. In that case it's also possible to calculate the scale and translation differences by using NuiImageGetColorPixelCoordinatesFromDepthPixel once. So no manual calibration is needed.

Hmm interesting, will let you know and share when I got it working

gabor_papp

Re: Kinect SDK block

6 months ago

You can also try aligning the two images using OpenCV stereo camera calibration. You need a chessboard pattern to show to your camera. To be able to take an image snapshot with the ir camera, you have to cover the ir projector for the calibration process and use some light so the ir camera can see the pattern. Having these pairs of images OpenCV can calibrate your cameras and you can align one surface to the other. I tried this with libfreenect, and it works quite well. Maybe even better then the automatic alignment of OpenNI in some cases.

-Gabor

timgroene..

Reageren: Kinect SDK block

6 months ago

Got it working now! Thanks to BanTheRewind's push in the right direction :)

It actually proved to be really easy.

What I need to do is render a texture where the color ( red and green) of each pixel represents the xy depth corrected color pixel. I just need to do this once. Like this ( in a situation where the depth resolution is 320x240 ) :

if( !mRenderedCorrectionMap ){
int width = 640;
int height = 480;
for( int x = 0 ; x < width; x++ ){
for( int y = 0 ; y < height ; y++ ){
Vec2i pos = mKinect->getColorPixelCoordinateFromDepthPixel( Vec2i( x/2, y/2 ) );
mCorrectionMap.setPixel( Vec2i( x, y ), ColorA( pos.x/(float)width, pos.y/(float)height, 0.0f, 0.0f ) );
}
}
mRenderedCorrectionMap = true;
}

Then I can bind a shader where mCorrectionMap texture is used as a displacement map

tex0 is texture from the RGB camera

tex1 is the rendered correctionMap

uniform sampler2D tex0;
uniform sampler2D tex1;
void main()
{
vec4 colorDisplace = texture2D(tex1, gl_TexCoord[0].xy);
vec4 color = texture2D(tex0, vec2( colorDisplace.r, colorDisplace.g ) );
gl_FragColor = color;
}

Tim

edit : The mCorrectionMap surface needs to be Surface32f

letsgoout..

Reageren: Kinect SDK block

6 months ago

Nicely done. Will work something along these lines into the block. Thinking I could both wrap the getColorPixelCoordinateFromDepthPixel method, as well as introduce a convenience method to generate the correction map.

http://www.BanTheRewind.com

timgroene..

Reageren: Kinect SDK block

6 months ago

I made a small adjustment to the method, enabling it to work with different resolutions.

ci::Vec2i Kinect::getColorPixelCoordinateFromDepthPixel( ci::Vec2i & input, NUI_IMAGE_RESOLUTION videoResolution, NUI_IMAGE_RESOLUTION depthResolution ){
long x,y;
mSensor->NuiImageGetColorPixelCoordinatesFromDepthPixelAtResolution( videoResolution, depthResolution, NULL, input.x, input.y, mDepthSurface.getPixel( input ).r, &x, &y);
return Vec2i( x, y );
}

You can get the result on git : http://github.com/TimGroeneboom/KinectEcard

Thanks for all the help :)

Tim

letsgoout..

Reageren: Kinect SDK block

6 months ago

Yeah, I'm actually going to have it use the working resolution of the device. This is how the getSkeletonDepthPos
and getSkeletonVideoPos methods work.

http://www.BanTheRewind.com

timgroene..

Re: Kinect SDK block

6 months ago

I actually found out that you can call the method with different resolutions then you're actually using. ( for example 1280x960 for the depth resolution, while it's actually running at 320x240 ), it will still return the correct displacement. This is useful for rendering a high resolution displacement map that make my video stream running at 1280x960 and perfectly aligned with the depth stream.

todd_9

Re: Kinect SDK block

6 months ago

Quick question about depth units...

If I get a depth value like this:

uint16_t depth = mDepthSurface.getPixel( Vec2i( pixNumber, lineNumber ) ).r;

How would I go about converting depth into real-world dimensions, in meters?

Thanks in advance.

timgroene..

Re: Kinect SDK block

6 months ago

Well, if I had to guess I would do something like this.

1. Get the range of your Kinect depth sensor. The closest point the sensor can detect is 800mm and furthest is 4000mm ( with near mode this is 400mm and 3000mm check : http://msdn.microsoft.com/en-us/library/hh438998.aspx ).

2. If black ( r = 0 ) represents the closest distance ( 800mm ) and white ( r = 255 ) represents the furthest distance ( 4000mm ).

Then getting the meter of that pixel will go something like this.

float far = 4.0f;
float near = 0.8f;
float depth = mDepthSurface.getPixel( Vec2i( pixNumber, lineNumber ) ).r / 255.0f
float meter = near + depth * ( far-near ) ;

gabor_papp

Re: Kinect SDK block

6 months ago

I think the depth information is not linear, so there are some approximations to do this, like this one by by Stéphane Magnenat:

float distance = 0.1236 * tan( depth / 2842.5 + 1.1863 );

in meters.

For more information:

http://openkinect.org/wiki/Imaging_Information#Depth_Camera

kazero

Re: Kinect SDK block

2 months ago

Hi, thanks for the block - looks great!

I have managed to run the samples in VS 2010 with Cinder 0.8.4 and everyhing works as expected. However I have problems running the samples in VS 2012.

I'm using Cinder 0.8.5.dev and the dev branch of Cinder-Kinect-SDK. I've build the Kinect.lib from source in VS 2012 and configured the sample project to use the v110 toolset. When I try to build it, I get some strange errors in glee.h

Error 1 error C1083: Cannot open include file: 'windows.h': No such file or directory c:\cinder\cinder_0.8.5\include\cinder\gl\glee.h 57 1 SkeletonApp

3 IntelliSense: cannot open source file "GL/gl.h" c:\Cinder\cinder_0.8.5\include\cinder\gl\GLee.h 60 2

Of course the files are there, as they are compiling just fine with v100

Probably it's something trivial, but how can I get it working?

Best Regards

num3ric

Re: Kinect SDK block

2 months ago

I am also getting this crash "windows.h"/glee crash, using the dev branch. Being a newbie with Cinder on Windows/VS2012, I'm not sure which steps to take next. Is the block not VS2012-ready yet, or is it simply an incorrect include path?

The non-D3D(?) Developer Toolkit v1.6.0 Kinect samples do work after their solution is updated so at least I know it's possible.

Éric

letsgoout..

Re: Kinect SDK block

2 months ago

It's not VS12 ready yet. There is a lot of progress on this, but I just haven't had time to post it yet. This weekend for sure. I've just wrapped up most of my major updates to the LeapSdk block I was giving priority.

http://www.BanTheRewind.com

kazero

Re: Kinect SDK block

2 months ago

By creating a new VS12 project from scratch and using the original code, I was able to get some of the examples working out of the box(Skeleton, Contours, PointCloud, SkeletonBitmap....others like Mesh, etc. don't work out of the box, but it's the Cinder API changes that break some of them.

Thanks for the block!

letsgoout..

Re: Kinect SDK block

2 months ago

I just pushed an update to the master branch with a few important changes:

VC11 projects added for all samples.
Tinderbox PointCloud template added.
The term "Video" has been replaced with "Color" (eg, addVideoCallback() is now addColorCallback() )
New method, ci::Vec2i Kinect::getColorDepthPos( const ci::Vec2i& ) added. This returns the location of a pixel in the color image for any pixel in the depth image. You can use this for creating pixel-perfect color point clouds.
Kinect SDK 1.6 implemented.
OpenCV 2.4.3 implemented in ContoursApp sample.
AudioInputWin dropped from KinectApp.
Block is at version 1.6.0.
Dropped "All" solution.

As always, feedback is appreciated. Let me know if I missed anything in this update.

http://www.BanTheRewind.com

Top Reply

{"z5754748":[23286000001178003,23286000001178005,23286000001186001,23286000001190001],"z5321590":[23286000000926115,23286000000924181,23286000000926199],"z3906314":[23286000001365035],"z4120513":[23286000001145035,23286000001339161,23286000001337215],"z4040385":[23286000000927036],"z5423336":[23286000001373061,23286000001374267,23286000001374279,23286000001373099,23286000001375049,23286000001375067],"z4730534":[23286000001187041],"z4943358":[23286000001277019,23286000001270119],"z7217641":[23286000001262053,23286000001262057,23286000001264203,23286000001264215,23286000001271001,23286000001372209],"z3501876":[23286000001374273,23286000001374369],"z3943121":[23286000001486219],"z24672551":[23286000001484035,23286000001487153],"z6320206":[23286000001045102,23286000001047016,23286000001048003,23286000001058013],"z18833200":[23286000001240085,23286000001238051,23286000001250003,23286000001249059,23286000001251005,23286000001271005],"z3492827":[23286000001262033,23286000001267111],"z21697295":[23286000001364101,23286000001362269,23286000001362411,23286000001362417,23286000001361173],"z5184985":[23286000001337145,23286000001338295],"z4600682":[23286000001277023],"z4746548":[23286000001199027,23286000001195115,23286000001227047,23286000001233005,23286000001235005,23286000001341025,23286000001341027],"z5032509":[23286000000926019],"z21751387":[23286000001364133,23286000001364135,23286000001364137],"z5682698":[23286000001142003],"z3493023":[23286000000927014,23286000000926049,23286000000924107,23286000000925032,23286000000924089,23286000000924109,23286000000925118,23286000000926197,23286000000926207,23286000000932019,23286000000931015,23286000000932027,23286000000942033,23286000000941053,23286000001043104,23286000001043110,23286000001057031,23286000001134293,23286000001138031,23286000001145019,23286000001145161,23286000001153023,23286000001179001,23286000001185001,23286000001188003,23286000001190007,23286000001188055,23286000001196005,23286000001197033,23286000001215005,23286000001229109,23286000001232017,23286000001240089,23286000001238057,23286000001249005,23286000001266011,23286000001267099,23286000001267101,23286000001266015,23286000001264241,23286000001272009,23286000001276055,23286000001278011,23286000001337151,23286000001339107,23286000001338415,23286000001338453,23286000001343013,23286000001361141,23286000001364143,23286000001364141,23286000001372071,23286000001374297,23286000001374323,23286000001487135,23286000001487191],"z8375123":[23286000001141005],"z5085115":[23286000000926011,23286000000926090],"z3908723":[23286000000926043,23286000000924087],"z4163224":[23286000001280001],"z2968461":[23286000000943011]}

Response title

Attachments

Replies(126)

Statistics

Tags

Actions

Related Posts