Learning to Follow and Generate Instructions for Language-Capable Navigation

Abstract

Visual-language navigation (VLN) is a challenging task that requires embodied agents to follow natural language instructions to navigate in previously unseen environments. However, existing literature put most emphasis on interpreting instructions into actions, only delivering “dumb” wayfinding agents which cannot actively use natural language to communicate with humans. In this article, we devise <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Lana</small> , a <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><u>la</u>nguage-capable</b> <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</u> avigation <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a</u> gent which is able to not only execute human-written navigation commands, but also provide route descriptions to humans. This is achieved by simultaneously learning instruction following and generation with only one single model. More specifically, two encoders, respectively for route and language encoding, are built and shared by two decoders, respectively, for action prediction and instruction generation, so as to exploit cross-task knowledge and capture task-specific characteristics. Throughout pretraining and fine-tuning, both instruction following and generation are set as optimization objectives. We further extend <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Lana</small> by exploiting object semantics during route encoding. This leads to <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Lana+</small> , a more powerful framework that simulates the way humans refer to landmarks for instructions composition and wayfinding. We empirically verify that, compared with recent advanced task-specific solutions, <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Lana</small> attains better performances on both instruction following and generation, with nearly half complexity. In addition, endowed with language generation capability, <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Lana</small> can explain to humans its behaviors and assist human's wayfinding. Benefiting from landmark information, <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Lana+</small> exhibits even more impressive performance. This work is expected to foster future efforts towards building more trustworthy and socially-intelligent navigation robots.