Multimodal Fabric Defect Classification Using Channel Switching and Multiscale Feature Fusion

Abstract

Human perception relies on the important sensory modes of vision and touch, which provide complementary information about the surrounding environment. Robots can leverage this multimodal sensing capability. In this study, we propose a defect recognition and classification method based on multiscale and channel-switching fusion, integrating visual and tactile modalities to achieve superior performance. Defect features are extracted by processing tactile data from GelSight sensors and captured visual images through deep neural networks. However, due to the heterogeneity between the two modalities and the diversity in defect sizes and shapes, commonly used methods including feature map superposition or concatenation operations increase computational costs and reduce model classification efficiency. To address this challenge, we designed a channel-switching module and a multiscale fusion module. The channel-switching module assigns different weights to each channel in the feature map to solve the problem of mutual exclusivity or unbalanced fusion in traditional methods, while the multiscale fusion module effectively processes features of different shapes by merging features from different scales. Results obtained on a newly collected fabric dataset show that the proposed defect classification method improves classification accuracy compared to learning from single-modal data, with significant improvement in complex backgrounds. The proposed fusion network framework achieves robust defect classification performance exceeding 90%.