Visual Manipulation Relationship Detection with Fully Connected CRFs for Autonomous Robotic Grasp

Abstract

In multi-object scenes, objects may be stacked and interact with each other, which brings an enormous difficulty for robotic grasping tasks. Therefore, exploring the manipulation relationships between objects is necessary. In robotic fields, there have been some works focusing on this task. However, most of them only detect the relationship between each pair of objects, regardless of dependency among them. In this paper, we construct a fully connected Conditional Random Fields (CRFs) on the output of front-end network, which models the dependency among all relationships in a scene. Besides, an exact inference algorithm and a variational inference algorithm are deployed for the CRFs, which immensely improves the performance of our framework while meeting the real-time requirements. Furthermore, we expand the types of visual manipulation relationship, which make the description pattern more powerful and stable. Our experiments show that the proposed approach gets a state-of-the-art result on Visual Manipulation Relationship Dataset (VMRD). Finally, based on this work, we build a robotic system for multi-object grasping, which demonstrates the practicality of our algorithm.