Catch It! Learning to Catch in Flight with Mobile Dexterous Hands

Abstract

Catching objects in flight (i.e., thrown objects) is a common daily skill for humans, yet it presents a significant challenge for robots. This task requires a robot with agile and accurate motion, a large spatial workspace, and the ability to interact with diverse objects. In this paper, we build a mobile manipulator composed of a mobile base, a 6-DoF arm, and a 12-DoF dexterous hand to tackle such a challenging task. We propose a two-stage reinforcement learning framework to efficiently train a whole-body-control catching policy for this high-DoF system in simulation. The objects' throwing configurations, shapes, and sizes are randomized during training to enhance policy adaptivity to various trajectories and object characteristics in flight. The results show that our trained policy catches diverse objects with randomly thrown trajectories, at a high success rate of about 80 % in simulation, outperforming the baselines significantly. The policy trained in simulation can be deployed seamlessly to the real world with only onboard sensing and computation, which achieves catching sandbags in various shapes, randomly thrown by humans. Our video and code are available at https://mobile-dex-catch.github.io.