Joint Objective and Subjective Fuzziness Denoising for Multimodal Sentiment Analysis

摘要

Multimodal Sentiment Analysis (MSA) aims at teaching computers or robotics to understand human sentiment with diverse multimodal signals, including audio, vision, and text. Current MSA approaches primarily concentrate on devising fusion strategies for multimodal signals and trying to learn better multimodal joint representations. However, employing multimodal signals directly is not appropriate since the human psychological states are fuzzy and can not be categorized easily, which undermines the effectiveness of existing methods. In this paper, we regard the natural fuzziness of human sentiments can be observed as two types: objective fuzziness introduced by human expression and subjective fuzziness caused by the complexity of human affection. Based on the assumption, we proposed a novel method termed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Joint Objective and Subjective Fuzziness Denoising (JOSFD)</i> , which introduced fuzzy logic into the multimodal fusion process and sentiment decision process to overcome the objective and subjective fuzziness. Specifically, our JOSFD method contains two key modules: (1) Modality-Specific Fuzzification Module leveraging uncertainty estimation and fuzzy logic to overcome the influence of objective fuzziness in different modalities in multimodal fusion. (2) Attitude-Intensity Representation Disentangling that learns joint representations for human attitude and sentiment strength separately and further employs fuzzy logic to decide the sentiment analysis results. We evaluate our proposed JOSFD method on three widely used MSA benchmark datasets, CMU-MOSI, CMU-MOSEI, and CH-SIMS. Extensive experiments demonstrate our proposed JOSFD method outperforms recent state-of-the-art methods.