Improving user interfaces of interactive robots with multimodality

Abstract

Interacting with robots like service robots should be as natural as possible for humans. As a consequence, such robots should be equipped with a multimodal user interface, so that interaction can rely on the relevant modalities for a given task and a given context. Typically, a rather fixed combination of speech and graphical user interface is used both for input and output. To enable more flexibility in the design of the interfaces and in the choice of modalities, we propose using an existing platform for model-driven generation of human-computer interfaces. In this paper, we present how the concepts of the platform apply to multimodal human-robot interaction as well, and we present an extension of the platform in this direction. To validate our approach, we investigated a simple user interface enabling the command of a mobile robot using dynamic gesture and speech. We provide some quantitative results of the performance of the fusion, which highlight not only the feasibility of the concept but also an interesting complementary of gesture and speech on the chosen task.