AI as Modality in Human Augmentation: Toward New Forms of Multimodal Interaction with AI-Embodied Modalities

Abstract

AI techniques have always been central to multimodal interaction, being leveraged for sensing user intention, action, and emotion and for generating adaptive feedback. In this process, AI has played the role of enabling technology toward making interactions with computer systems and computerized environments more efficient, accessible, and natural through the prism of multimodality. In this work, we argue that AI can play yet another key role in multimodal interaction by implementing distinctive modalities complementing those accessible by humans on their own. The result is multimodal interaction through human-AI integration, where multimodality is distributed across users and AI embodiments of various form factors, from robotic assistants to smart wearables to ambient devices. The new concept of “AI as modality,” both contrasting and complementing the established “AI as tool,” is particularly relevant for recent interaction paradigms involving human-computer integration and human augmentation. In this paper, we outline the vision of leveraging AI for delivering distinctive modalities toward the next generation of multimodal interactive experiences.