Language-guided Semantic Mapping and Mobile Manipulation in Partially\n Observable Environments

Abstract

Recent advances in data-driven models for grounded language understanding\nhave enabled robots to interpret increasingly complex instructions. Two\nfundamental limitations of these methods are that most require a full model of\nthe environment to be known a priori, and they attempt to reason over a world\nrepresentation that is flat and unnecessarily detailed, which limits\nscalability. Recent semantic mapping methods address partial observability by\nexploiting language as a sensor to infer a distribution over topological,\nmetric and semantic properties of the environment. However, maintaining a\ndistribution over highly detailed maps that can support grounding of diverse\ninstructions is computationally expensive and hinders real-time human-robot\ncollaboration. We propose a novel framework that learns to adapt perception\naccording to the task in order to maintain compact distributions over semantic\nmaps. Experiments with a mobile manipulator demonstrate more efficient\ninstruction following in a priori unknown environments.\n