Grounding Language Instructions that Refer to the Past, Present, and/or Future State of Objects for Human-Robot Interaction

摘要

For robots to effectively collaborate with human partners, they need to be able to understand what instructions and/or statements mean in the context of their environment. To ground objects in a dynamic world, robots need an understanding of how both spatial and temporal relationships evolve as a function of time in the environment. However, it is computationally intensive to classify, track, and predict the motion of all objects. Approaches based on Language-Guided Temporally Adaptive Perception (LGTAP) utilize information embedded in the instruction to selectively classify objects to construct minimal but sufficiently detailed models of the environment for symbol grounding. Such methods, however, fail when the instruction refers to the future state of the environment as it lacks any notion of whether to and for how long a future prediction is necessary to ground the instruction. This prompts a reformulation of LGTAP that can selectively utilize information from past observations to accurately predict the future state of objects. This paper describes a novel approach for LGTAP for instructions that may refer to the past, present, and/or future state of the environment by closing the loop around symbol grounding and adaptive perception. A detailed analysis of a grounding problem that refers to the future state of the environment, a corpus-based analysis of performance, and a physical demonstration of natural language understanding is presented along with a description of this novel architecture.