Situated Language Understanding at 25 Miles per Hour

Abstract

In this paper, we address issues in situ-ated language understanding in a rapidly changing environment – a moving car. Specifically, we propose methods for un-derstanding user queries about specific tar-get buildings in their surroundings. Unlike previous studies on physically situated in-teractions such as interaction with mobile robots, the task is very sensitive to tim-ing because the spatial relation between the car and the target is changing while the user is speaking. We collected situated utterances from drivers using our research system, Townsurfer, which is embedded in a real vehicle. Based on this data, we analyze the timing of user queries, spa-tial relationships between the car and tar-gets, head pose of the user, and linguis-tic cues. Optimized on the data, our al-gorithms improved the target identification rate by 24.1 % absolute. 1