Towards an integrated model of speech and gesture production for multi-modal robot behavior

Abstract

The generation of communicative, speech-accompanying robot gesture is still largely unexplored. We present an approach to enable the humanoid robot ASIMO to flexibly produce speech and co-verbal gestures at run-time, while not being limited to a pre-defined repertoire of motor actions. Since much research has already been dedicated to this challenge within the domain of virtual conversational agents, we build upon the experience gained from the development of a speech and gesture production model used for the virtual human Max. We propose a robot control architecture building upon the Articulated Communicator Engine (ACE) that was developed to allow virtual agents to flexibly realize planned multi-modal behavior representations on the spot. Our approach tightly couples ACE with ASIMO's perceptuo-motor system, combining conceptual representation and planning with motor control primitives for speech and arm movements of a physical robot body. First results of both gesture production and speech synthesis using ACE and the MARY text-to-speech system are presented and discussed.