Multimodal human-robot interaction in an assistive technology context

Abstract

In this paper, we present a prototype robotic system that captures, processes and fuses speech, vision and laser-depth data to more accurately interpret and perform simple tasks in a domestic environment. We can never assume that any one of these inputs are completely accurate, but by using a combination, a more accurate interpretation could be found. For each speech, gesture recognition and object recognition input, an associated probability of correctness is calculated. Using these probabilities, a predefined associative map between specific words and gestures, and contextual information, we update the associated probability until some threshold has been reached. This contextual information includes the history of recently uttered phrases, as well as the information about known objects in the environment. Once an interpretation, complete or otherwise, is found, a response is formulated. This system has been developed using a Mitsubishi Heavy Industries robot, Wakamaru, as a platform 1.