MMF-YOLO-based intelligent coal gangue grabbing robotic arm under different illumination with multi-modal sensors

Abstract

In the coal mining, coal gangue can easily obstruct subsequent processing and transportation. However, variations in lighting conditions in real-world environments pose significant challenges to the stability and accuracy of traditional visual systems. Additionally, the coal gangue has a highly similar background to the surrounding coal, and its shape is often irregular, which further complicates the task for visual detection systems. To address these challenges, we propose a novel object detection method MMF-YOLO for coal gangue detection under varying lighting conditions. First, we construct a dual-branch network architecture for visible and infrared images. In addition, a Multi-model Fusion (MMF) module is proposed to fuse features from two brunches. By combining the advantages of both visible and infrared images, the MMFYOLO model can extract more robust features in different lighting environments. To enhance the distinction between coal gangue and background, a Spatial Context Enhancement Module (SCEM) is proposed, which integrates global background information to strengthen feature extraction. Finally, we incorporate laser ranging and design an intelligent robotic arm for fluctuating lighting conditions. Experimental results show that the proposed method significantly improves the accuracy and robustness of coal gangue detection. Compared to the baseline YOLOv8, the mAP of MMF-YOLO improves by 7.2%.