Automatic recognition of engagement and emotion in a group of children

摘要

Young children are developing social skills at their own pace, large variances in development occasionally occur and lead to imbalanced engagement in small groups. In that situation, an agent such as a social robot can recognise individual levels of engagement and support a child who is less engaged than others. This dissertation aimed to develop automatic methods for the recognition of engagement levels and emotion of children in a small group play setting. This posed great challenges such as modeling temporal and group dynamics in a noisy environment. My thesis consists of three parts: (1) automatic ranking of engagement in a group of children; (2) automatic recognition of emotion in speech (SER); and (3) integration of emotional states into an engagement-ranking model. Related to (1), firstly, I created a corpus of spontaneous interactions of children in group play and an ordinal coding scheme. Secondly, I examined various machine learning methods including a pairwise ranking approach, and found the pairwise ranking approach worked best for the engagement ranking task. Thirdly, I proposed a temporal ranking algorithm that uses a pairwise ranking model and probabilistic transition model. To develop (2), I aggregated multiple corpora and developed multi-task learning-based methods that better generalize speech emotion recognition models over variations in contextual factors such as speaker’s gender and naturalness of emotional expressions. Moreover, I examined if skip-connections can ease optimization of deep temporal architectures. Finally, I proposed three-dimensional convolutional network based methods that model spectral-temporal dynamics without expensive temporal memory units. To achieve (3), I developed an emotional-state-based feature set by using SER methods. I also proposed a Deep Convolutional Ranking Network (DCRN) that learns discriminative representations of features for pairwise relations between engagement levels. I found the emotional feature set performed best and DCRN significantly outperformed a conventional method. Throughout these three parts, I developed automatic methods that recognise individual engagement levels by considering the temporal and group dynamics of engagement. My results indicated that the ordinal coding and learning was the most suitable for the task. This dissertation also provides practical and rigorous guidelines of evaluating deep architectures.