Incremental Referent Grounding with NLP-Biased Visual Search

Abstract

Human-robot interaction poses tight timing require-ments on visual as well as natural language processing in order to allow for natural human-robot interaction. In particular, humans expect robots to incrementally resolve spoken references to visually perceivable objects as the referents are verbally described. In this pa-per, we present an integrated robotic architecture with novel incremental vision and natural language process-ing and demonstrate that incrementally refining atten-tional focus using linguistic constraints achieves signif-icantly better performance of the vision system com-pared to non-incremental visual processing.