FFT / OpenCV Capture Bug

in Using Cinder • 1 year ago

Hey all - apologies in advance if this question is poorly worded. I'm a bit new to Cinder and extremely rusty with C++ and have run into a bug which has me completely stumped.

Basically, I'm taking OpenCV Capture data, processing it using an Optical Flow algorithm and using the data alongside FFT information from an audio file to feed a visualization. There seems to be some conflict between my Update method and the DrawFFT method I'm using. It is almost as if the pointers or allocation of the Audio Buffer and Capture Input arrays are being mixed up. The weird thing is the app crashes either at the Update method or the FFT method randomly. If I comment out all Update code other than the Audio Buffer input line, the app works fine.

If anyone has any ideas I'd be grateful!

Update Crash BT:

#0 0x0025c06f in cinder::ImageSource::rowFuncSourceRgb<unsigned char, unsigned char, (cinder::ImageIo::ColorModel)1, false> (this=0x289fea0, target=@0xbfffdcc0, row=337, data=0x1009d760) at ImageIo.cpp:193
#1 0x0018bdf3 in cinder::ImageSourceSurface::load (this=0x289fea0, target=@0xbfffdd58) at Surface.cpp:81
#2 0x00055832 in cinder::ChannelT<unsigned char>::ChannelT (this=0xbfffe0a0, imageSource=@0xbfffe098) at Channel.h:164
#3 0x0000b720 in MyApp::update (this=0x1a81400) at /Users/praveen/Documents/XCodeProjects/MyApp/xcode/MyApp.cpp:348
#4 0x00029723 in cinder::app::App::privateUpdate__ (this=0x1a81400) at App.cpp:155
#5 0x0002f191 in -[AppImplCocoaBasic timerFired:] (self=0xe83980, _cmd=0x99242917, t=0x28867c0) at AppImplCocoaBasic.mm:111
#6 0x9b34dbdf in __NSFireTimer ()
#7 0x9bc1d656 in __CFRUNLOOP_IS_CALLING_OUT_TO_A_TIMER_CALLBACK_FUNCTION__ ()
#8 0x9bc1cfe7 in __CFRunLoopDoTimer ()
#9 0x9bbfbf70 in __CFRunLoopRun ()
#10 0x9bbfb47c in CFRunLoopRunSpecific ()
#11 0x9bbfb328 in CFRunLoopRunInMode ()
#12 0x9345017f in RunCurrentEventLoopInMode ()
#13 0x934574e7 in ReceiveNextEventCommon ()
#14 0x93457356 in BlockUntilNextEventMatchingListInMode ()
#15 0x987dba9c in _DPSNextEvent ()
#16 0x987db306 in -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] ()
#17 0x987d7675 in -[NSApplication run] ()
#18 0x0002fed6 in -[AppImplCocoaBasic init:] (self=0xe83980, _cmd=0x5b4e5a, aApp=0x1a81400) at AppImplCocoaBasic.mm:64
#19 0x0008f787 in cinder::app::AppBasic::launch (this=0x1a81400, title=0x5af178 "MyApp", argc=3, argv=0xbffff940) at AppBasic.cpp:95
#20 0x0002a28c in cinder::app::App::executeLaunch (app=0x1a81400, renderer=0xe834e0, title=0x5af178 "MyApp", argc=3, argv=0xbffff940) at App.cpp:473
#21 0x0000f67f in cinder::app::AppBasic::executeLaunch (app=0x1a81400, renderer=0xe834e0, title=0x5af178 "MyApp", argc=3, argv=0xbffff940) at AppBasic.h:197
#22 0x0000d9d5 in main (argc=3, argv=0xbffff940) at /Users/praveen/Documents/XCodeProjects/MyApp/xcode/MyApp.cpp:549

Draw FFT Crash BT:

#0 0x00159189 in cinder::audio::PcmBufferT<float>::getSampleCount (this=0x2d68050, channelId=cinder::audio::CHANNEL_FRONT_LEFT) at PcmBuffer.h:73
#1 0x0000c5cd in MyApp::drawFft (this=0x1332a00) at /Users/praveen/Documents/XCodeProjects/MyApp/xcode/MyApp.cpp:432
#2 0x0000c513 in MyApp::draw (this=0x1332a00) at /Users/praveen/Documents/XCodeProjects/MyApp/xcode/MyApp.cpp:389
#3 0x00028662 in cinder::app::App::privateDraw__ (this=0x1332a00) at App.cpp:173
#4 0x00095868 in -[CinderView drawRect:] (self=0x255cb40, _cmd=0x9918f658, rect={origin = {x = 0, y = 0}, size = {width = 640, height = 480}}) at CinderView.mm:116
#5 0x9882b295 in -[NSView _drawRect:clip:] ()
#6 0x9885ac32 in -[NSView _recursiveDisplayAllDirtyWithLockFocus:visRect:] ()
#7 0x98828bf2 in -[NSView _recursiveDisplayRectIfNeededIgnoringOpacity:isVisibleRect:rectIsVisibleRectForView:topView:] ()
#8 0x9882387c in -[NSView _displayRectIgnoringOpacity:isVisibleRect:rectIsVisibleRectForView:] ()
#9 0x9881c732 in -[NSView displayIfNeeded] ()
#10 0x989007d3 in -[NSView display] ()
#11 0x000958d4 in -[CinderView draw] (self=0x255cb40, _cmd=0x9913ba0c) at CinderView.mm:105
#12 0x0002f1e2 in -[AppImplCocoaBasic timerFired:] (self=0x255a730, _cmd=0x99242917, t=0xe79e40) at AppImplCocoaBasic.mm:112
#13 0x9b34dbdf in __NSFireTimer ()
#14 0x9bc1d656 in __CFRUNLOOP_IS_CALLING_OUT_TO_A_TIMER_CALLBACK_FUNCTION__ ()
#15 0x9bc1cfe7 in __CFRunLoopDoTimer ()
#16 0x9bbfbf70 in __CFRunLoopRun ()
#17 0x9bbfb47c in CFRunLoopRunSpecific ()
#18 0x9bbfb328 in CFRunLoopRunInMode ()
#19 0x9345017f in RunCurrentEventLoopInMode ()
#20 0x934574e7 in ReceiveNextEventCommon ()
#21 0x93457356 in BlockUntilNextEventMatchingListInMode ()
#22 0x987dba9c in _DPSNextEvent ()
#23 0x987db306 in -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] ()
#24 0x987d7675 in -[NSApplication run] ()
#25 0x0002ff06 in -[AppImplCocoaBasic init:] (self=0x255a730, _cmd=0x5b4e7a, aApp=0x1332a00) at AppImplCocoaBasic.mm:64
#26 0x0008f7b7 in cinder::app::AppBasic::launch (this=0x1332a00, title=0x5af198 "MyApp", argc=3, argv=0xbffff940) at AppBasic.cpp:95
#27 0x0002a2bc in cinder::app::App::executeLaunch (app=0x1332a00, renderer=0x25677f0, title=0x5af198 "MyApp", argc=3, argv=0xbffff940) at App.cpp:473
#28 0x0000f6cf in cinder::app::AppBasic::executeLaunch (app=0x1332a00, renderer=0x25677f0, title=0x5af198 "MyApp", argc=3, argv=0xbffff940) at AppBasic.h:197
#29 0x0000da25 in main (argc=3, argv=0xbffff940) at /Users/praveen/Documents/XCodeProjects/MyApp/xcode/MyApp.cpp:546

Here are the two methods which seem to be causing the issue. If I comment out all Update code other than the Audio Buffer line, the app never crashes:

UPDATE & DRAW FFT Methods

void MyApp::update()

{

if(mTrack && mTrack->isPcmBuffering()) {

//get the latest pcm buffer from the track

mPcmBuffer = mTrack->getPcmBuffer();

}



for( vector<Capture>::iterator cIt = mCaptures.begin(); cIt != mCaptures.end(); ++cIt ) {

int index = (cIt - mCaptures.begin());

Capture cap = mCaptures[index];

index = mCaptures.size() - (cIt - mCaptures.begin()) - 1;

if( cap.checkNewFrame() && !mVelocitys[index].empty() ) {

mSurfaces[index] = cap.getSurface();



//flow method always HS black and white:

//mTexture = gl::Texture( fromOcv( mVelocitys[index] ) );



cv::Mat currentFrame( toOcv( Channel( mSurfaces[index] ) ) );

if( mPrevFrames[index].data ) {

calculateFlow( currentFrame, mPrevFrames[index], mVelocitys[index],       mVelocityXs[index], mVelocityYs[index] );

cv::absdiff(currentFrame, mPrevFrames[index], mAbsDiffs[index]);

}

mCurrentFrames[index] = mPrevFrames[index];

mPrevFrames[index] = currentFrame;

}

}



if(mVelocitys.size() > 0) {

CvScalar velResult = cv::mean(mVelocitys[0]);

double vel = velResult.val[0] * 100;



CvScalar massResult = cv::mean(mAbsDiffs[0]);

double mass = (massResult.val[0] - 1.0);



int min = 0;

int max = 127;



normalizedVel = ((double)(vel - min) / (double)(max - min)) * 10000;

normalizedMass = ((double)(mass - min) / (double)(max - min)) * 100;



if(normalizedVel > 100.0) normalizedVel = 100.0;

if(normalizedMass > 100.0) normalizedMass = 100.0;

}

}
#if defined(CINDER_MAC)

void MyApp::drawFft()

{

float ht = 150.0f;

uint16_t bandCount = 128;

if (mPcmBuffer && mPcmBuffer->getChannelCount() > 0 && mPcmBuffer->getSampleCount() > 0) {



//use the most recent Pcm data to calculate the Fft

std::shared_ptr<float> fftRef = audio::calculateFft( mPcmBuffer->getChannelData( audio::CHANNEL_FRONT_LEFT ), bandCount );



if( ! fftRef ) {

return;

}



float *fftBuffer = fftRef.get();

float avgVolume = 0;

float avgBass = 0;



//draw the bands

for( int i = 0; i < ( bandCount ); i++ ) {

float barY = fftBuffer[i] / bandCount * ht;

if(!prevHt) prevHt = barY;



float newBarY = ((barY + prevHt) / 2) + 25;



prevHt = barY;



if(i < 4) { avgBass += barY; }

avgVolume += barY;



glBegin( GL_QUADS );

glColor3f( colorFromFloat(normalizedVel / 100) );

glVertex2f( i * 30, ht );

glVertex2f( i * 30 + 10, ht );

glColor3f( colorFromFloat(normalizedMass / 10) );

glVertex2f( i * 30 + 10, ht - newBarY );

glVertex2f( i * 30, ht - newBarY );

glEnd();

}



if(avgBass / 4.0 > 90) {

osc::Message bassMsg;

bassMsg.addFloatArg(avgBass / 4.0);

bassMsg.setAddress("NikeSXSWClub/OSC/Bass");

bassMsg.setRemoteEndpoint(host, port);

sender.sendMessage(bassMsg);



osc::Message ampMsg;

ampMsg.addFloatArg(avgVolume / bandCount);

ampMsg.setAddress("NikeSXSWClub/OSC/Amplitude");

ampMsg.setRemoteEndpoint(host, port);

sender.sendMessage(ampMsg);



osc::Message velMsg;

velMsg.addFloatArg(normalizedVel / 100);

velMsg.setAddress("NikeSXSWClub/OSC/Velocity");

velMsg.setRemoteEndpoint(host, port);

sender.sendMessage(velMsg);



osc::Message massMsg;

massMsg.addFloatArg(normalizedMass / 10);

massMsg.setAddress("NikeSXSWClub/OSC/Mass");

massMsg.setRemoteEndpoint(host, port);

sender.sendMessage(massMsg);



//fprintf(stderr,"BASS AMPLITUDE: %f & TOTAL AMPLITUDE: %f\n",(avgBass / 4.0), (avgVolume / bandCount));

}

}

}

#endif

Replies(2)

morgan.pa..

Re: FFT / OpenCV Capture Bug

1 year ago

Hey Praveen,

Is it one of those exec_bad_access errors? My number one reason for weird bugs is accidentally writing past the end of an array, which is really really easy to do in C, and just steamrollers over whatever memory comes next.

Another wild guess is that it's a thread safety issue. I'm not smart enough to be able to really tell whether these methods are both being called on the same thread, but it looks like they're both getting called from a run loop, which makes me suspect that they're both on the same thread.

-Morgan