Pointing and Commanding Gesture Recognition in 3D for Human-Robot Interaction

Abstract

This paper proposes and develops a Kinect-based pointing and commanding gesture recognition system in 3D for human-robot interaction. The system can be divided into three subsystems which are pointing gesture recognition, 3D dynamic gesture recognition and robot navigation. Kinect Skeletal Tracking is used to track the hand (palm), shoulder and elbow joints of a human in 3D coordinate space. The 3D coordinates of these joints are then used to detect pointing gestures and estimate the pointing direction. To detect dynamic hand gestures, the right hand (palm) joint is tracked and its 3D coordinate sequence is recorded. These coordinates are then geometrically translated by a reference point and normalized to form the feature vector that is fed into a Hidden Markov Model (HMM) based classifier for training and classification. For training the HMMs, Baum-Welch Learning algorithm is used. The system is trained and tested using 5-fold cross validation method using 500 gesture instances of 5 predefined gestures performed by 10 volunteers. The system achieves an overall accuracy of 94.4% in recognizing dynamic gestures. This project proposes and implements two separate algorithms for robot navigation using the recognized pointing and commanding gestures. A simple simulator along with a graphical user interface is also developed for testing the proposed interaction system and it works successfully.