Inferring Maps and Behaviors from Natural Language Instructions

Abstract

Abstract. Natural language provides a flexible, intuitive way for peo-ple to command robots, which is becoming increasingly important as robots transition to working alongside people in our homes and work-places. To follow instructions in unknown environments, robots will be expected to reason about parts of the environments that were described in the instruction, but that the robot has no direct knowledge about. However, most existing approaches to natural language understanding require that the robot’s environment be known a priori. This paper pro-poses a probabilistic framework that enables robots to follow commands given in natural language, without any prior knowledge of the environ-ment. The novelty lies in exploiting environment information implicit in the instruction, thereby treating language as a type of sensor that is used to formulate a prior distribution over the unknown parts of the environment. The algorithm then uses this learned distribution to in-fer a sequence of actions that are most consistent with the command, updating our belief as we gather more metric information. We evaluate our approach through simulation as well as experiments on two mobile robots; our results demonstrate the algorithm’s ability to follow naviga-tion commands with performance comparable to that of a fully-known environment. 1