Vision-and-Language Navigation: Interpreting visually-grounded\n navigation instructions in real environments

摘要

A robot that can carry out a natural-language instruction has been a dream\nsince before the Jetsons cartoon series imagined a life of leisure mediated by\na fleet of attentive robot helpers. It is a dream that remains stubbornly\ndistant. However, recent advances in vision and language methods have made\nincredible progress in closely related areas. This is significant because a\nrobot interpreting a natural-language navigation instruction on the basis of\nwhat it sees is carrying out a vision and language process that is similar to\nVisual Question Answering. Both tasks can be interpreted as visually grounded\nsequence-to-sequence translation problems, and many of the same methods are\napplicable. To enable and encourage the application of vision and language\nmethods to the problem of interpreting visually-grounded navigation\ninstructions, we present the Matterport3D Simulator -- a large-scale\nreinforcement learning environment based on real imagery. Using this simulator,\nwhich can in future support a range of embodied vision and language tasks, we\nprovide the first benchmark dataset for visually-grounded natural language\nnavigation in real buildings -- the Room-to-Room (R2R) dataset.\n