Error-Aware Imitation Learning from Teleoperation Data for Mobile\n Manipulation

Abstract

In mobile manipulation (MM), robots can both navigate within and interact\nwith their environment and are thus able to complete many more tasks than\nrobots only capable of navigation or manipulation. In this work, we explore how\nto apply imitation learning (IL) to learn continuous visuo-motor policies for\nMM tasks. Much prior work has shown that IL can train visuo-motor policies for\neither manipulation or navigation domains, but few works have applied IL to the\nMM domain. Doing this is challenging for two reasons: on the data side, current\ninterfaces make collecting high-quality human demonstrations difficult, and on\nthe learning side, policies trained on limited data can suffer from covariate\nshift when deployed. To address these problems, we first propose Mobile\nManipulation RoboTurk (MoMaRT), a novel teleoperation framework allowing\nsimultaneous navigation and manipulation of mobile manipulators, and collect a\nfirst-of-its-kind large scale dataset in a realistic simulated kitchen setting.\nWe then propose a learned error detection system to address the covariate shift\nby detecting when an agent is in a potential failure state. We train performant\nIL policies and error detectors from this data, and achieve over 45% task\nsuccess rate and 85% error detection success rate across multiple multi-stage\ntasks when trained on expert data. Codebase, datasets, visualization, and more\navailable at https://sites.google.com/view/il-for-mm/home.\n