Hand-over-Face Gesture based Facial Emotion Recognition using Deep Learning

Abstract

Facial emotion recognition is significant for several applications such as human-computer interaction, video surveillance and robotics. Occlusion covering the face is a barrier in clearly recognizing facial emotion. Hence, in early researches, the hand-over-face occlusion is either removed or ignored while recognizing emotion. However, hand-over-face (hand gesture) along with facial expression represents an emotion. The existing hand-over-face gesture based emotion recognition methods identify merely basic emotions due to coding schema with limited hand gestures. In this paper, we propose a new hand-over-face gesture based emotion recognition method that includes coding schema with more hand gestures and uses deep learning. Specifically, we use Convolutional Neural Network (CNN) that eliminates the need for manual feature extraction and extracts more class specific features automatically. Moreover, we use Recurrent Neural Network (RNN) to recursively learn the features and classify them into more advanced emotion categories. Hence, our proposed method identifies more advanced emotions such as confident, making decision, scared, ashamed and ok sign along with basic emotions.