Gesture Correctness Estimation with Deep Neural Networks and Rough Path Descriptors

Abstract

One of the classical problems in computer vision is the automatic identification of different gestures or motions from a sequence of frames, captured via cameras or sensors, that usually contain angles or positions of many body joints. A large body of research addressing this problem exists in the literature. However, an interesting variant is not just to identify the gesture, but to decide whether a certain gesture is correctly or incorrectly executed. This can be in the context of physical exercises for rehabilitation in healthcare, robotics, sports, sign language and more. In this work we present our research on the automatic classification of gestures into correct or incorrect categories using the following three different but powerful techniques: convolutional neural networks (CNNs), long short-term memory (LSTM) neural networks and a technique based on rough path theory to extract unique sequences (i.e. path signatures) from gesture data and to use them as inputs to classical classifiers such as RandomForest (RF) and k-Nearest Neighbor (kNN). We collected the data we used as part of a collaborative project from Perkeso Tun Razak Rehabilitation Centre, Malacca, Malaysia. We recorded 17 patients performing nine types of gestures. For controls, we recorded 14 healthy individuals performing the same gestures. Our cross subject evaluation yields high accuracy for several subjects and gestures. CNNs and LSTMs achieve accuracy values up to 100% for several subjects in many gestures with the highest average accuracy for some gestures reaching 92%. The best overall classifier when using the rough paths representation was kNN as its average accuracy can reach 96.74% (100% accuracy for a subset of subjects in some gestures). For the three methods, the average accuracy across all gestures is 85.24%, 85.63% and 94.67% respectively. Overall, our research shows a great potential in identifying the right learning algorithm in motion correctness identification for real data.