Bio-Inspired Multisensory Fusion for Autonomous Robots

Abstract

Multimodal sensory fusion is a fundamental requirement for autonomous robots to form an unambiguous and meaningful representation of their surroundings. In this paper, we propose a multisensory self-organizing neural architecture for multimodal fusion using unsupervised machine learning. Inspired by biological evidence from the organization of the human sensory system, the proposed architecture consists of self-organizing neural layers for learning individual modalities. We have incorporated scalable computing for self-organization, so the processing can be scaled to support large datasets and short computation times. The lateral associative connections capture the co-occurrence relationships across individual modalities for cross-modal fusion in obtaining a multimodal representation. Experiments are conducted on an audio-visual dataset consisting of utterances to evaluate the quality of multimodal fusion over individual unimodal representations. Multimodal representation achieves significant improvements over the unimodal representations. These results indicate the proposed architecture is capable of forming effective multimodal representations in short computation times from congruent multisensory stimuli.