Robot Hearing Through Optical Channel in a Cocktail Party Environment

Abstract

The cocktail party problem refers to a challenging process when the human sensory system tries to separate a specific voice from a loud mixture of background sound sources. The problem is much more demanding for machines and has become the holy grail in robotic hearing. Despite many advances in noise suppression, the intrinsic information from the contaminated acoustic channel remains difficult to recover. Herein, a simple‐yet‐powerful laser‐assisted audio system termed robot ear accomplished by laser (REAL) is shown to probe the vibrations of sound‐carrying surfaces (mask, throat, and other nearby surfaces) in optical channel, which is intrinsically immune to acoustic background noises. The results demonstrate that REAL can directly obtain the audio‐frequency content from the laser without acoustic channel interference. The signals can be further transcribed into human‐recognizable audios by exploiting the internal time and frequency correlations through memory‐enabled neural networks. The REAL system would enable a new way in human–robot interaction. An interactive preprint version of the article can be found at: https://www.authorea.com/doi/full/10.1002/aisy.202200143 .