Speakers Raise Their Hands and Head During Self-Repairs in Dyadic Conversations

Abstract

People often encounter difficulties in building shared understanding during everyday conversation. The most common symptom of these difficulties are self-repairs, when a speaker restarts, edits, or amends their utterances mid-turn. Previous work has focused on the verbal signals of self-repair, i.e., speech disfluences (filled pauses, truncated words and phrases, and word substitutions or reformulations), and computational tools now exist that can automatically detect these verbal phenomena. However, face-to-face conversation also exploits rich nonverbal resources and previous research suggests that self-repairs are associated with distinct hand movement patterns. This article extends those results by exploring head and hand movements of both speakers and listeners using two motion parameters: 1) height (vertical position) and 2) 3-D velocity. The results show that speech sequences containing self-repairs are distinguishable from fluent ones: speakers raise their hands and head more (and move more rapidly) during self-repairs. We obtain these results by analyzing data from a corpus of 13 unscripted dialogs, and we discuss how these findings could support the creation of improved cognitive artificial systems for natural human–machine and human–robot interaction.