Combining CNN and LSTM for activity of daily living recognition with a 3D matrix skeleton representation

摘要

Abstract In socially assistive robotics, human activity recognition plays a central role when the adaptation of the robot behavior to the human one is required. In this paper, we present an activity recognition approach for activities of daily living based on deep learning and skeleton data. In the literature, ad hoc features extraction/selection algorithms with supervised classification methods have been deployed, reaching an excellent classification performance. Here, we propose a deep learning approach, combining CNN and LSTM, that exploits both the learning of spatial dependencies correlating the limbs in a skeleton 3D grid representation and the learning of temporal dependencies from instances with a periodic pattern that works on raw data and so without requiring an explicit feature extraction process. These models are proposed for real-time activity recognition, and they are tested on the CAD-60 dataset. Results show that the proposed model behaves better than an LSTM model thanks to the automatic features extraction of the limbs’ correlation. “New Person” results show that the CNN-LSTM model achieves $$95.4\%$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mn>95.4</mml:mn><mml:mo>%</mml:mo></mml:mrow></mml:math> of precision and $$94.4\%$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mn>94.4</mml:mn><mml:mo>%</mml:mo></mml:mrow></mml:math> of recall, while the “Have Seen” results are $$96.1\%$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mn>96.1</mml:mn><mml:mo>%</mml:mo></mml:mrow></mml:math> of precision and $$94.7\%$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mn>94.7</mml:mn><mml:mo>%</mml:mo></mml:mrow></mml:math> of recall.