Multi-Modal Hierarchical Empathetic Framework for Social Robots With Affective Body Control

Abstract

Social robots require the ability to understand human emotions and provide affective and behavioral responses during human-robot interactions. However, current social robots lack empathy capabilities. In this work, we propose a novel Multi-modal Hierarchical Empathetic (MHE) framework for generating empathetic responses for social robots. MHE is composed of a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multi-modal fusion and emotion recognition</i> module, an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">empathetic dialogue generation</i> module, and an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">expression generation</i> module. By fusing the sensor signals of different modalities, the robot can recognize human emotions and generate affective responses. Multiple experiments are conducted on a real robot, Pepper, to evaluate the proposed framework. The experiments are conducted to discriminate between MHE-generated text and human responses in complete ignorance, and most experimenters agree that MHE can effectively generate human-like and empathetic responses. To better evaluate the similarity between human-robot and human-human interactions, a period eye movement map (PEM) captured by an eye tracker is proposed. The experimental results demonstrate the improvement in the MHE in human-robot interactions by comparing different PEMs.