A Persistent Spatial Semantic Representation for High-level Natural\n Language Instruction Execution

Abstract

Natural language provides an accessible and expressive interface to specify\nlong-term tasks for robotic agents. However, non-experts are likely to specify\nsuch tasks with high-level instructions, which abstract over specific robot\nactions through several layers of abstraction. We propose that key to bridging\nthis gap between language and robot actions over long execution horizons are\npersistent representations. We propose a persistent spatial semantic\nrepresentation method, and show how it enables building an agent that performs\nhierarchical reasoning to effectively execute long-term tasks. We evaluate our\napproach on the ALFRED benchmark and achieve state-of-the-art results, despite\ncompletely avoiding the commonly used step-by-step instructions.\n