As a part of my work, I am trying to correlate the audio signal in a video with the pixels of each frame. The steps I follow are:

1] Audio sampling rate and frame rate of the video are known. So, division of the audio sampling rate with the frame rate gives us the number of audio samples associated with each frame of the video.

2] That being done, I want to correlate the audio signal (the collection of audio signal amplitudes with 2 channels) with the 2-D pixel intensity of each frame, to basically decipher whether which part of the frame has maximum correspondence with the audio.

3] But, obviously the 2 signals ( audio signal with 2 channels and the 2-D frame pixel intensity values ) are having different dimensions. In this scenario, how do I compute the correlation between them?

I did encounter the idea of performing canonical correlation, but then again, the unequal dimensions of the 2 signals makes it a problem.

Some insight in this regard would be much helpful.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.