[Cryptography] "The Visual Microphone: Passive Recovery of Sound from Video"

[Christian Huitema]

>> Very nice demo, but it requires a really high speed camera -- Nyquist and
>> all that. 
> 
> Did you watch through to the end?  Starting at around 3 minutes in, they
show 
> how to use artifacts due to shutter motion to recover sound reasonably
well using 
> a standard camera - frequencies 5 times the frame rate of the camera.

Yes I saw that. What they call a standard camera is in fact a rolling
shutter DLSR camera. The rolling shutter causes each line to be captured at
slightly different time. They get a "super-sampling" effect from analyzing
the movement of pixels on each line of the camera. That would absolutely not
work with a standard webcam, in which all the pixels are captured at the
same time.

>From the paper: "We took a video of a bag of candy (Fig. 10(a)) near a
loudspeaker playing speech, and took a video from a viewpoint orthogonal to
the loudspeaker-object axis, so that the motions of the bag due to the
loudspeaker would be horizontal and fronto-parallel in the camera’s image
plane. We used a Pentax K-01 with a 31mm lens. The camera recorded at 60 FPS
at a resolution of 1280×720 with an exposure time of 1/2000 seconds. By
measuring the slope of a line, we determined it to have a line delay of 16µs
and a frame delay of
5 milliseconds, so that the effective sampling rate is 61920 Hz with 30% of
the samples missing. The exposure time caps the maximum recoverable
frequency at around 2000 Hz." 

The technique that they describe effectively gets 1280 pixels per line and
compute their vibration on the horizontal axis. This is what they allude to
in the "viewpoint orthogonal to the loudspeaker-object axis." If the
loudspeaker had been placed in a direction perpendicular to the lines, they
would not have captured anything. Of course, that could be mitigated by
having two cameras at a 90 degrees angle. But at that point, you are
probably better off building a specialized device.

-- Christian Huitema