Late-Fusion Multimodal Human Detection Based on RGB and Thermal Images for Robotic Perception

Abstract

This paper addresses the problem of detecting humans in RGB and Thermal (long-wave IR) images taken by cameras mounted onboard a mobile robot. Human/Pedestrian detection is currently one of the most pertinent object detection problems, mainly due to safety concerns in autonomous vehicles. The majority of approaches apply deep-learning techniques based solely on RGB images. However, they have a few shortcomings, namely that during foggy weather, nighttime, and low-light scenarios, these images may not contain sufficient information. To address these issues, this work studies the use of thermal cameras as a complementary source of information for human detection in indoor and outdoor environments. The proposed approach uses YOLOv5 to detect pedestrians in both thermal and RGB images. Moreover, the different modalities are combined using early and late fusion techniques. Evaluation of the proposed approach is carried out in the FLIR Aligned dataset and in a new in-house dataset. Results indicate that the use of fusion techniques highlights a promising way to improve the overall performance in this application domain.