Multimodal Emotion Recognition for Human Robot Interaction

Abstract

Emotion recognition is one of the most important assets that humans possess, which helps them to communicate with each other in the most efficient manner. Humans recognize each other's emotions through facial features, audio features and body gestures. Limited work is reported in literature, wherein multiple modalities are considered for emotion recognition. In this work, models are trained on facial and speech samples using 1-Dimensional (1D) and 2-Dimensional (2D) Convolutional Neural Networks (CNN) and also trained using pretrained networks such as Visual Geometry Group-16 (VGG-16) and Inception Version 3 (V3). Suitable features are extracted from face and speech to classify the emotions. Analysis of speech features is performed to highlight the difference between the expressiveness based on genders. For the final combined model both facial and speech models are trained separately and combined based on a decision tree, which depends on the individual model's test case accuracy.