Towards augmenting dialogue strategy management with multimodal sub-symbolic context

Abstract

Abstract. A synthetic agent requires the coordinated use of multiple sensory and effector modalities in order to achieve a social human-robot interaction (HRI). While systems in which such a concatenation of multiple modalities exist, the issue of information coordination across modalities to identify relevant context information remains problematic. A system-wide information formalism is typically used to address the issue, which requires a re-encoding of all information into the system ontology. We propose a general approach to this information coordination issue, focussing particularly on a potential application to a dialogue strategy learning and selection system embedded within a wider architecture for social HRI. Rather than making use of a common system ontology, we rather emphasise a sub-symbolic association-driven architecture which has the capacity to influence the ‘internal ’ processing of all individual system modalities, without requiring the explicit processing or interpretation of modality-specific information.