Fine Tuning Vision Transformer Model for Facial Emotion Recognition: Performance Analysis for Human-Machine Teaming

Abstract

Facial Emotion Recognition (FER) has become essential in various domains, including robotic systems, affective computing, emotion-triggered intelligent agents, and human-computer interaction for human-machine teaming. Although Convolutional Neural Network (CNN)-based models were popular for facial emotion classification, Transformer-based models have shown better performance in computer vision tasks such as image classification, semantic segmentation, and object detection. In this study, we explore the performance of the Vision Transformer model on a publicly available large FER dataset called AffectNet, which provides a realistic representation of emotions “in the wild.” We fine-tuned the model for the emotion classification task based on facial expressions. We achieved an accuracy of 64.48% on the Affectnet validation set, outperforming many other methods that use only transformer models. Further, we explore how they can be used for Human-Machine Teaming particularly in vehicular systems to improve driver safety, comfort, and experience.