Attention-Based Multimodal Fusion for Estimating Human Emotion in Real-World HRI

Abstract

Toward empathetic and harmonious human-robot interaction (HRI), automatic estimation of human emotion has attracted increasing attention from multidisciplinary research fields. In this report, we propose an attention-based multimodal fusion approach that explores the space between traditional early and late fusion approaches, to deal with the problem of asynchronous multimodal inputs while considering their relatedness. The proposed approach enables the robot to align the human's visual and speech signals (more specifically, facial, acoustic, and lexical information) extracted by its cameras, microphones, and processing modules and is expected to achieve robust estimation performance in real-world HRI.