Multiple Roles of Multimodality Among Interacting Agents

Abstract

The term multimodality has come to take on several somewhat different meanings depending on the underlying theoretical paradigms and traditions along with the purpose and context of use. The term is closely related to embodiment , which, in turn, is also used in several different ways. In this article, we elaborate on this connection and propose that a pragmatic and pluralistic stance is appropriate for multimodality. We further propose a distinction between first- and second-order effects of multimodality—what is achieved by multiple modalities in isolation and the opportunities that emerge when several modalities are entangled. This highlights questions regarding ways to cluster or interchange different modalities, for example, through redundancy or degeneracy. Apart from discussing multimodality with respect to an individual agent, we further look to more distributed agents and situations in which social aspects become relevant. In robotics, understanding the various uses and interpretations of these terms can prevent miscommunication when designing robots as well as increase awareness of the underlying theoretical concepts. Given the complexity of the different ways in which multimodality is relevant in social robotics, this can provide the basis for negotiating appropriate meanings of the term on a case-by-case basis.