Creating Multimodal Interactive Agents with Imitation and\n Self-Supervised Learning

Abstract

A common vision from science fiction is that robots will one day inhabit our\nphysical spaces, sense the world as we do, assist our physical labours, and\ncommunicate with us through natural language. Here we study how to design\nartificial agents that can interact naturally with humans using the\nsimplification of a virtual environment. We show that imitation learning of\nhuman-human interactions in a simulated world, in conjunction with\nself-supervised learning, is sufficient to produce a multimodal interactive\nagent, which we call MIA, that successfully interacts with non-adversarial\nhumans 75% of the time. We further identify architectural and algorithmic\ntechniques that improve performance, such as hierarchical action selection.\nAltogether, our results demonstrate that imitation of multi-modal, real-time\nhuman behaviour may provide a straightforward and surprisingly effective means\nof imbuing agents with a rich behavioural prior from which agents might then be\nfine-tuned for specific purposes, thus laying a foundation for training capable\nagents for interactive robots or digital assistants. A video of MIA's behaviour\nmay be found at https://youtu.be/ZFgRhviF7mY\n