Vocal Interaction with a 7-DOF Robotic Arm for Object Detection, Learning and Grasping

Abstract

This work presents preliminary results on the de- velopment of a system for multimodal interaction between a user and a robotic arm for human-robot cooperation tasks. In particular the system allows the user to ask the robot to grasp objects lying on a tabletop in the robot's working space using natural language. The objects are detected using a low-cost RGB-D camera. The robot is able to deal with fundamental issues such as multiple object instances and unknown objects. In the first case, the robot asks the user for further information on which particular object instance to pick. In the latter case, the robot asks the user to point to the unknown object to be learned. The robot communicates with the user using speech synthesis with natural sentences. The object detection pipeline is robust to partial occlusions. The system has been developed using ROS and was evaluated on a small 7-DOF anthropomorphic arm with a set of household items.