Real-time virtual UR5 robot imitation of human motion based on 3D camera

Abstract

Human robot interaction is playing significant roles in many industries, in which humans are allowed to corporate with robots or control robots to complete some tasks. Achieving realtime control of robots by demonstrating human arm motion is one of the effective methods to improve the quality of human robot interaction. In this paper, a new method to accomplish human skeleton key points extraction is proposed and a system for controlling a virtual UR5 robot (which is modeling in Unity 3D) is developed to imitate human motions. A sequential architecture composed of convolutional networks is used to identify the 2D coordinates; the depth information from Kinect v2 sensor is used to obtain the actual 3D coordinates and the result of the proposed method is used for joint angles calculation. The appropriate mapping relationships between human arm and UR5 robots are built by transforming the arm angel to UR5's degrees of freedom. The performance of this method is measured by the degree of real-time simulation of human hand trajectory. Further, the comparison between the method in this paper and other method is implemented. The experimental results indicate that the developed system and proposed method are reliable and efficient to accomplish human motion imitation.