Kinect-Based Hand Tracking for First-Person-Perspective Robotic Arm Teleoperation

Abstract

This paper deals with the man-machine interaction of robotic arm teleoperation by the developed Kinect and first-person-perspective follow-up technologies. Kinect is used to collect and preprocess the depth information to determine the hand position vector. With the help of the virtual robotic arm set up by Processing, the relationship between the hand and shoulder is mapped to the one between the robotic arm's chassis and its gripper, then it can be output to the Arduino lower computer, so that the corresponding motion of the robotic arm is controlled. The first-person-perspective follow-up obtains the position information of the head by the positioning chip fixed on the VR glasses. After data transmission and processing, the angles of pan-tilt platform, i.e., the pitch and yaw angle of the camera, are changed to keep the orientation of the camera consistent with the user’s eyes. The VR glasses provide real-time view of the first-person perspective. Compared with the existing ones, the proposed tele-operating robotic arm enhanced by Kinect and binocular camera has the advantages of better immersion experience and intuitive operation.