Millimeter-Wave Radar and RGB-D Camera Sensor Fusion for Real-Time People Detection and Tracking

Abstract

One of the key aspects of modern-day robotics research is the development of perceptual capabilities of agents. Robots need to understand their surroundings in order to reason or infer about a given situation. Chief among the areas of perception is that of detection and tracking of people. In this work we employ millimeter-wave radar and imaging sensor fusion approach to pedestrian detection and tracking. We perform experiments in a variety of settings (single and multi- target, varying illumination, varying distances and fields of view, dense and light clutter, and through-the-wall tracking). Our results show that our fusion and tracking architecture is far superior to camera only systems in terms of accuracy and added functionality. Our implementation mitigates the effects of occlusions (including wooden walls), blurry images, obscured lens, and field of view limitations.