Specific hand motion patterns correlate to miscommunications during dyadic conversations

Abstract

Effective and natural communication is achieved by exchanging several multi-modal signals through highly coordinated communication mechanisms. These mechanisms are frequently subject to troubles of speaking in the form of disfluencies, typically followed by a self-repair from the speaker (i.e. to try to fix the misunderstanding): overall, these are signs of a possible miscommunication. Automatically detecting miscommunications is crucial to implement conversational agents, either digital or robotic, that could successfully interact with people. This can be done by searching for specific patterns across different communication channels, for example disfluencies in the speech signal or specific movements of the limbs. However, what are the motion patterns that correlate to miscommunications is still unclear. In this paper we report a human study in which we identify one of such patterns: in particular, we show that the hands of the speaker reliably move upwards during miscommunications. We performed a statistical analysis of synchronized speech and motion tracking data extracted from natural conversations of 15 dyads; our results show a statistically significant tendency of moving hands upwards during speech disfluencies, which are a clear sign of miscommunication.