Semantic Localization Via the Matrix Permanent

Abstract

Most approaches to robot localization rely on lowlevel geometric features such as points, lines, and planes. In this paper, we use object recognition to obtain semantic information from the robot's sensors and consider the task of localizing the robot within a prior map of landmarks, which are annotated with semantic labels. As object recognition algorithms miss detections and produce false alarms, correct data association between the detections and the landmarks on the map is central to the semantic localization problem. Instead of the traditional vectorbased representations, we use random finite sets to represent the object detections. This allows us to explicitly incorporate missed detections, false alarms, and data association in the sensor model. Our second contribution is to reduce the problem of computing the likelihood of a set-valued observation to the problem of computing a matrix permanent. It is this crucial transformation that enables us to solve the semantic localization problem with a polynomial-time approximation to the set-based Bayes filter. The performance of our approach is demonstrated in simulation and in a real environment using a deformable-part-model-based object detector. Comparisons are made with the traditional lidarbased geometric Monte-Carlo localization.