Learning Structured World Models From and For Physical Interactions

Abstract

Humans have a strong intuitive understanding of the physical world. Through observations and interactions with the environment, we build mental models that predict how the world would change if we applied a specific action (i.e., intuitive physics). My research draws on these human insights to develop model-based RL agents that learn from their interactions and build predictive models that generalize widely across a range of objects made with different materials. The core idea behind my research is to introduce novel representations and integrate structural priors into learning systems to model dynamics at different levels of abstraction. I will discuss how such structures can make model-based planning algorithms more effective, helping robots accomplish complex manipulation tasks (e.g., manipulating an object pile, shaping deformable foam into a target configuration, and making a dumpling from dough using various tools).