Real time vision-based human hand motion tracking and grasping for arobotic manipulator with soft hand

Abstract

Abstract: This paper presents a vision-based framework for the 7 degree-of-freedom (7-DOF) manipulator to imitate human hand movements for pick-and-place tasks. Using two cameras, Google’s MediaPipe is used to extract 21 key points of the human hand in each frame and generate three-dimensional (3D) coordinates of the hand. Robot Operation System (ROS) architecture is used to integrate the hardware together. The robotic system can imitate the user’s actions and achieve the goal of object pick-and-place based on the translation, rotation, and grasping of the hand. Experimental studies are carried out to demonstrate the effectiveness of the proposed work.