Multi-modal Visual-Thermal Saliency-based Object Detection in Visually-degraded Environments

Abstract

In this work we present a new approach on the fusion of multi-modal data from visible-light and thermal camera sensors in order to efficiently detect objects in visually-degraded environments using aerial robots. Through a bottom-up approach inspired by the principles of visual saliency and how it models the attention mechanism of mammals, a novel algorithm that builds individual conspicuity maps across color channels and across visible-light and longwave infrared spectra is proposed. More specifically, two input images - a radiometric thermal image and a visible-light RGB image - are fed respectively into an intensity channel, and a set of intensity and color-opponent channels. For every channel, image pyramids are computed to allow searching in different scales. The eventual feature maps are subsequently combined into one saliency map. The approach is computationally efficient and tailored to the limited processing capacities found onboard Micro Aerial Vehicles, while at the same time providing fast real-time object detections as required for the purposes of autonomous navigation and object search. Integrating thermal vision together with visible-light data, the method builds on top of a rich representation that allows resilient operation even in conditions of visual degradation and specifically in cases of low-light or obscurant-filled (e.g., dust, fog) settings which can be penetrated using thermal vision. The presented algorithm and overall system-level solution is built to facilitate efficient, fast and reliable object detection during autonomous aerial robotic exploration of subterranean environments, contributing novel capacity beyond the context of aerial robotic navigation. Our approach is extensively evaluated in the framework of real-life field deployments in underground metal mines in Northern Nevada.