<title>Integrating task-directed planning with reactive object recognition</title>

Abstract

We describe a robot vision system that achieves complex object recognition with two layers of behaviors, performing the tasks of planning and object recognition, respectively. The recognition layer is a pipeline in which successive stages take in images from a stereo head, recover relevant features, build intermediate representations, and deposit 3-D objects into a world model. Each stage is an independent process that reacts automatically to output from the previous stage. This reactive system operates continuously and autonomously to construct the robot's 3-D model of the environment. Sitting above the recognition pipeline is the planner which is responsible for populating the world model with objects that satisfy the high-level goals of the system. For example, upon examination of the world model, the planner can decide to direct the head to another location, gating new images into the recognition pipeline, causing new objects to be deposited into the world model. Alternatively, the planner can alter the recognition behavior of the pipeline so that objects of a certain type or at a certain location appear in the world model.