GesNavi: Gesture-guided Outdoor Vision-and-Language Navigation

Abstract

Vision-and-Language Navigation (VLN) task involves navigating mobility using linguistic commands and has application in developing interfaces for autonomous mobility.In reality, natural human communication also encompasses non-verbal cues like hand gestures and gaze.These gesture-guided instructions have been explored in Human-Robot Interaction systems for effective interaction, particularly in object-referring expressions.However, a notable gap exists in tackling gesturebased demonstrative expressions in outdoor VLN task.To address this, we introduce a novel dataset for gesture-guided outdoor VLN instructions with demonstrative expressions, designed with a focus on complex instructions requiring multi-hop reasoning between the multiple input modalities.In addition, our work also includes a comprehensive analysis of the collected data and a comparative evaluation against the existing datasets.