Recognition of Bimanual Manipulation Categories in RGB-D Human Demonstration

Abstract

Humans exhibit outstanding capabilities in using both hands to perform daily tasks. Understanding bimanuality in human demonstrations is key for humanoid robots, which should learn from human observation. In this paper, we address the problem of the recognition and segmentation of bimanual action categories defined by our Bimanual Manipulation Taxonomy, based on RGB-D data. To this end, we combine object detection and human motion tracking methods to derive graph-based representations of bimanual manipulation tasks that describe spatial relations between objects and hands as well as the temporal change of these relations during the execution of the task. We train a Graph Neural Network (GNN) for simultaneous recognition and segmentation of the demonstrations and compare the results with a rule-based classification approach that only takes contact relations between objects and hands into account. For training, five kitchen tasks of the KIT Bimanual Actions Dataset are selected and complemented with two new tasks accounting for symmetrical bimanual manipulations. The evaluations show the best results for a GNN considering spatial relations and object knowledge compared to a GNN considering only contact relations between objects and hands and compared to the rule-based approach.