[Cryptography] "The Visual Microphone: Passive Recovery of Sound from Video"

[Christian Huitema]

> Everything leaks.  Film a bag of chips through a soundproof window, decode
its 
> vibrations to listen to what's being said in the room.
> 
> https://www.youtube.com/watch?v=FKXOucXB4a8
> 
> I'll bet you could go a pretty good job of decoding keyclicks from a
"secure" room this way.

Very nice demo, but it requires a really high speed camera -- Nyquist and
all that. 

The beauty there is the "passive" part. The active version has been out
there for a long time. Point a laser at an object that could potentially
vibrate, measure the intensity of the returned light, feed the vibrations to
a loud speaker. Apart from the laser, the electronics are really
straightforward -- you could describe the capture system as a "single pixel
camera."

They use something far more sophisticated than a single pixel camera, as
explained in
http://people.csail.mit.edu/mrub/papers/VisualMic_SIGGRAPH2014.pdf: "Video
frame rates are in the range of 2kHz-20kHz, with resolutions ranging from
192x192 pixels to 700x700 pixels. Sounds were played at loud volumes ranging
from 80 dB (an actor’s stage voice) to 110 dB (comparable to a jet engine at
100 meter)... Processing each video typically took 2 to 3 hours using MATLAB
on a machine with two 3.46GHz processors and 32GB of RAM." They need a
relatively high pixel definition to retrieve the minute motions induced by
the sound waves, and they need the high frame rate to beat Nyquist. Multiply
high definition by high frame rate and you get a really large bandwidth
requirement. 

On the other hand, bandwidth requirement and computing requirement will only
be a problem for a limited time. Just light crypto, computer vision
algorithm only get better with time. I can foresee someone implementing the
basic filtering steps in an FPGA right inside a special purpose camera,
drastically reducing the bandwidth requirement. Somebody else will probably
use machine learning techniques to train better filters. What is a lab
curiosity now could become a viable product in a short time!

-- Christian Huitema