Teach robots understanding new object types and attributes through natural language instructions

Abstract

Robots often have limited knowledge about the environment and need to continuously acquire new knowledge in order to collaborate with the humans. To address this issue, this paper presents a method which allows the human to teach a robot new object types and attributes through natural language (NL) instructions. A simple yet robust vision algorithm is proposed to segment objects and describe the relations between objects. The segmented objects as well as their relations are regarded as the basic knowledge of the robot. The NL instructions are processed to domain-specific representations for the robot to identify the target objects. The target objects as well as the object type or attribute labels referred in the NL instructions are collected as training samples for the robot to learn. Experimental results demonstrate the effectiveness and advantages of the proposed method.