DexH2R: Task-Oriented Dexterous Manipulation From Human to Robots

Abstract

Dexterous manipulation remains a challenging problem due to the high-dimensional action space of robot hands. To solve this challenge, recent advances focus on transferring human hand motions to dexterous robot motions. However, the most straightforward method, retargeting, fails to ensure successful task execution due to the ignorance of hand–object interaction and environmental feedback, while other methods combining retargeting with human feedback significantly increase human labor and degrade motion quality. To address this limitation, we propose a novel framework that enhances retargeted motions with a residual policy based on reinforcement learning, allowing robots to autonomously correct action errors without human intervention. Specifically, our approach leverages human demonstrations to provide natural and task-relevant finger motions while incorporating object and environmental information to improve manipulation performance. Extensive experiments in multiple robot hand embodiments demonstrate that our method outperforms baseline approaches by approximately 40% in task success rates on both seen and unseen objects. Furthermore, our approach eliminates the need for real-time human correction, significantly reducing labor costs and facilitating scalable data collection for dexterous manipulation.