Multimodal spatio-temporal framework for real-world affect recognition

摘要

Deep learning models show great potential in applications involving video-based affect recognition, including human-computer interaction, robotic interfaces, stress and depression assessment, and Alzheimer's disease detection. The low complex Multimodal Diverse Spatio-Temporal Network (MDSTN) has been analysed to effectively capture spatio-temporal information from audio-visual modalities for affect recognition using the Acted Facial Expressions in the Wild (AFEW) dataset. The scarcity of data is handled by data augmented parallel feature extraction for visual network. Visual features extracted by carefully reviewing and customizing Convolutional 3D architecture over different ranges are combined to train a neural network for classification. Multi-resolution Cochleagram (MRCG) features from speech, along with spectral and prosodic audio features, are processed by a supervised classifier. The late fusion technique is explored to integrate audio and video modalities, considering their processing over different temporal spans. The MDSTN approach significantly boosts the accuracy of basic emotion recognition to 71.54% on the AFEW dataset. It demonstrates exceptional proficiency in identifying emotions such as disgust and surprise, thus exceeding current benchmarks in real-world affect recognition. • To efficiently represent spatio-temporal information from audio-visual modalities linked to affect recognition, a novel lightweight parallel augmented Multimodal Diverse Spatio-Temporal Network (MDSTN) is implemented. • The proposed architecture is a blend of deep learning and machine learning. • The fundamental building blocks of MDSTN are various receptive fields that aim to explore a wide range of multimodal temporal dynamics. • MDSTN explores the potential for audio and visual modalities to complement each other in dynamic natural environments. The scarcity of data is handled by data augmented parallel feature extraction for visual network. • Multi-resolution Cochleagram features (MRCG) from speech are merged with spectral and prosodic audio features and input into a 1D CNN, whilst visual features are extracted using a parallelly augmented customised C3D architecture. • The integration of audio and visual information considers their processing over various temporal spans and is achieved by a weighted score-level fusion approach. • The cross-corpus analysis shows robustness of model on diverse data. • By identifying emotions such as disgust and surprise, MDSTN method considerably enhances the accuracy of basic emotion detection to 71.54% on AFEW dataset and improved discrimination power for each emotion. An increase in the F1-score signifies a substantial improvement in the model’s classification performance, exceeding current benchmarks in real-world affect recognition.