Unified robot learning of action labels and motion trajectories from 3D human skeletal data

Abstract

Currently, robot learning of human activities is mainly studied in two largely disconnected domains: high level semantics understanding in human activity recognition, and low level motion trajectory reproduction in robot imitation learning. The critical problem of human activity unified learning (HAUL) was not well studied in previous work. One important challenge is the lack of a representation that can be learned from both levels. We introduce a novel approach to address this HAUL problem at the representation level, by simultaneously learning action labels and motion trajectories from publicly available 3D human skeletal datasets, thus avoiding additional human labor for data collection. Our approach builds a subject and body position independent shared skeleton, and extracts features of skeletal activities based on this model. Then the extracted features are encoded by the parameter set of Gaussian Mixture Models to construct the unified representations. The proposed compact representation can be directly applied to identify activity labels when combined with Support Vector Machines, and can be also employed to generate trajectories of the learned activities when combined with Gaussian Mixture Regression on a robot. Finally, an inverse kinematic mapping is developed to transfer human skeletal trajectories to joint angle sequences in the robot's embodiment. Empirical studies using simulation and real humanoid robots demonstrate that our approach achieves promising performance on robot unified learning of human action labels and motion trajectories, effectively addressing the HAUL problem.