Person tracking and gesture recognition in challenging visibility conditions using 3D thermal sensing

Abstract

Many existing person tracking systems are challenged by non-laboratory scenarios, including variable lighting conditions, rain, smoke, tracking distance, and tracking speed. We provide evidence that by using a 3D thermal sensor, a person can be tracked in three dimensions with high success using very simple tracking methods, in many of the challenging lighting conditions and other weather conditions that confound other systems. In support of our claim, we present the PROWL (Perception for Robotic Operation over Widespread Lighting) sensor system, which uses thermal stereo image processing and on-board sensor processing to perform person tracking and gesture recognition. PROWL, using only ICP-based point matching algorithms, obtains 100% person tracking success at 20 frames per second out to 13 meters and zero false-positive/false-negative gesture recognition within 7 meters in all tested scenarios, which includes a sunny outdoor environment, a nighttime outdoor environment, a blackout indoor environment, and a whiteout smoke-filled indoor environment.