Learning Affordance Segmentation for Real-World Robotic Manipulation via Synthetic Images

Abstract

This letter presents a deep learning framework to predict the affordances of object parts for robotic manipulation. The framework segments affordance maps by jointly detecting and localizing candidate regions within an image. Rather than requiring annotated real-world images, the framework learns from synthetic data and adapts to real-world data without supervision. The method learns domain-invariant region proposal networks and task-level domain adaptation components with regularization on the predicted domains. A synthetic version of the UMD data set is collected for autogenerating annotated, synthetic input data. Experimental results show that the proposed method outperforms an unsupervised baseline, and achieves performance close to state-of-the-art supervised approaches. An ablation study establishes the performance gap between the proposed method and the supervised equivalent (30%). Real-world manipulation experiments demonstrate use of the affordance segmentations for task execution, which achieves the same performance with supervised approaches.