Modelling daily actions through hand-based spatio-temporal features

Abstract

In this paper, we propose a new approach to domestic action recognition based on a set of features which describe the relation between poses and movements of both hands. These features represent a set of basic actions in a kitchen in terms of the mimics of the hand movements, without needing information of the objects present in the scene. They address specifically the intra-class dissimilarity problem, which occurs when the same action is performed in different ways. The goal is to create a generic methodology that enables a robotic assistant system to recognize actions related to daily life activities and then, be endowed with a proactive behavior. The proposed system uses depth and color data acquired from a Kinect-style sensor and a hand tracking system. We analyze the relevance of the proposed hand-based features using a state-space search approach. Finally, we show the effectiveness of our action recognition approach using our own dataset.