Deep Multimodal Fusion with Corrupted Spatio-Temporal Data Using Fuzzy Regularization

Abstract

Deep networks have been successfully applied to unsupervised feature learning and supervised classification and regression for unimodal data (e.g., sensors, images, or audio). Multimodal data is often used to improve the performance of networks according to the slogan: the more, the better. Limited research is available to compensate for corrupted signals from multimodal approaches. In this work, we propose a novel regularization method for deep networks to learn features over multiple modalities designed to compensate for relative sensor weaknesses, such as sensor malfunction, inaccuracy, restricted spatial coverage, and uncertainty. We have used a special augmentation strategy for image and time series modalities to enhance the dataset of underrepresented industrial failure cases. The primary objective is to prevent these cases from negatively impacting the model's predictions. Our approach involves incorporating a fuzzy regularizer that can modify the intensity of activations depending on the quality of the signal, enabling disturbances from various modalities to be identified based on the activations. Our experiments on a simulated Universal Robots UR5 dataset demonstrate the effectiveness of our proposed regularization in increasing the model's stability, accuracy, and generalization to uncertainties and failures in the input signals. By incorporating fuzzy regularization in deep fusion models, their efficiency on complex tasks can be improved while reducing the complexity of the architectures.