What no robot has seen before — Probabilistic interpretation of natural-language object descriptions

Abstract

We investigate the task of recognizing objects of daily use in human environments purely based on object descriptions given in natural language. In particular, we present an approach to transform phrases stated in natural language that describe such objects by their visual appearance into formal, semantic representations of their perceptual characteristics, which in turn can be used in a robot perception system in order to identify objects that the robot has never encountered before. To this end, we learn probabilistic first-order knowledge bases from encyclopedic articles and online dictionaries, which contain textual descriptions of a vast amount of everyday objects. We demonstrate the applicability of the approach on a robotic system in a proof-of-concept evaluation on a selected set of object descriptions acquired from the internet.