Being able to extract audio from a soundless visual recording is something that intelligence agencies will likely one day use – if they don’t already do it – for their operations. Researchers from MIT, Microsoft and Adobe have managed to create an algorithm that can reconstruct audio “by analyzing minute vibrations of objects depicted in video,” like bags of chips found in the same room with a target, as well as other objects including aluminum foil, the surface of a glass of water, and the leaves of a plant.
The researchers looked at the unique vibrations different sounds make after recording them on high-frame rate videos of a room through sound proof glass. The researchers achieved their best results if their camera’s frame rate was higher than the frequency of the audio signal, although commercial cameras capturing 60fps were also successfully used.
The naked eye won’t capture the actual movements of an object, as the sound waves make them move only a small fraction of a pixel. However, when the video is then analyzed by a computer, which captures all the movements of an object when hit by sound waves, the sound can be recomposed fairly well. Recovered sound is so good in one experiment that Shazam is able to recognize the music played while the video was recorded, which was reconstructed from the object’s multiple vibrations.
“This is new and refreshing. It’s the kind of stuff that no other group would do right now,” associate professor of electrical engineering and computer science at the University of California at Berkeley Alexei Efros said about the new discovery. “We’re scientists, and sometimes we watch these movies, like James Bond, and we think, ‘This is Hollywood theatrics. It’s not possible to do that. This is ridiculous.’ And suddenly, there you have it. This is totally out of some Hollywood thriller. You know that the killer has admitted his guilt because there’s surveillance footage of his potato chip bag vibrating.”
A video explaining this amazing, and scary, new technology follows below.
