What Do Foundation Models have to Do With and For HRI?

Abstract

Foundation models have unlocked major advancements in AI. What do foundation models have to do with and for Human-Robot Interaction - And how can HRI help unlock more powerful foundation models for robot learning and embodied reasoning? In this talk, I will discuss examples of how foundation models could enable a step function in human-robot interaction research, including: how to leverage foundation models to enable multimodal human-robot communication, enable non-expert users to teach robots new low-level skills and personalized high level plans through natural interactions, create new expressive robot behaviors, etc. At the same time, foundation models still have significant gaps in human-robot interaction contexts. I will share early insights showing that HRI could be key to evolving the foundation models themselves, enabling even more powerful interactions, and improving robot learning. The fields of HRI and robot learning seem to have evolved and grown in parallel, but foundation models might be the breakthrough we needed to bring these fields together. Now there is a unique opportunity for HRI to unlock robot learning in-the-wild, not only because it will yield robots that are more useful and adaptable to humans, but because it will enable improving the foundation models that will likely affect every aspect of robot learning.