Facial Expression Recognition Based on Zero-Addition Pretext Training and Feature Conjunction-Selection Network in Human–Robot Interaction

Abstract

The design of the feature extraction process and training strategy are crucial aspects of achieving high-performance facial expression recognition (FER). Although the introduction of low-dimensional texture descriptors is advantageous for extracting facial expression features, most of the hybrid feature-based methods need data preprocessing phase, which is not conducive to the deployment in human–robot interaction (HRI) systems. Moreover, some FER studies use facial recognition datasets to pretrain the network, but they fail to effectively capture the facial action features that are specific to FER. In this article, we proposed a zero-addition pretext training and feature conjunction-selection network (FCSNet) for FER in HRI. The zero-addition pretext training task is the eye aspect ratio (EAR)-level recognition (ELR), which improves the perception of ocular muscles by designing the training strategy. FCSNet consists of two main components: LBP-conjunction convolution that improves the surface facial expression representation by integrating LBP filters into the convolution layer and semantic feature selection mechanism (SFSM) that filters out irrelevant and ambiguous semantic information from high-level facial expression features. Our method achieved 97.93% and 83.80% recognition accuracy on radboud faces database (RaFD) and our collected testing set, respectively. We also conducted a preliminary application experiment to demonstrate the practicality of our method in HRI.