A Multimodal Approach for Emotion Recognition in Conversations Using the MELD Dataset

摘要

Emotion recognition in conversational contexts is a fundamental task in affective computing, with significant implications for applications in empathetic dialogue systems, social robotics, and advanced media analysis. Given the complexity of emotional expression, effective emotion recognition systems must integrate multiple modalities, including linguistic, acoustic, and visual cues. This paper presents a multimodal framework for emotion recognition, evaluated on the Multimodal EmotionLines Dataset (MELD) dataset, which consists of multiparty dialogues containing textual, acoustic, and visual data. The proposed system leverages a pretrained Bidirectional Encoder Representations from Transformers (BERT) model for text encoding, Mel-frequency cepstral coefficients (MFCCs) for acoustic features, and Residual Network-50 (ResNet-50) for extracting visual embeddings. These modality-specific features are then fused through a concatenation mechanism and passed through a linear classifier to predict one of seven emotion categories: neutral, joy, sadness, anger, surprise, fear, and disgust. Experimental results demonstrate that the proposed approach achieves a test accuracy of 73.56%, a weighted F1-score of 0.7254, outperforming unimodal baselines. Analysis of the confusion matrix indicates that the model excels at classifying common emotions, such as neutral and joy, but struggles with more subtle emotions, such as fear and disgust, which are often misclassified. Further evaluations, including ablation studies and comparisons with existing methods, underscore the efficacy of multimodal fusion. Future research will explore the integration of temporal attention mechanisms, context modeling across dialogue turns, and improved feature extraction strategies for more precise emotion recognition.