Hand Gesture-Guided Manipulator for Enhancing Human-Robot Collaboration

Abstract

Utilizing hand gestures to control robots offers several advantages over traditional command methods. Gestures provide a more intuitive interaction, improving their usability and flexibility. This study presents a hand gesture-guided robotic system for controlling the Universal Robot (UR3e) arm to enhance human-robot collaboration. This research aims to develop an efficient communication system between a user and the robot arm. A vision-based control system is proposed that integrates real-time hand gesture recognition via the MediaPipe framework. The system enables a camera assistant or assistant surgeon to intuitively adjust camera positioning—zoom, tilt, and rotate—without physical contact. The system was evaluated using a UR3e robotic arm with integrated and external cameras for hand gesture recognition and real-time visualization. The setup involved the robot arm’s camera capturing a test board with four target points. Evaluation results demonstrate high accuracy in gesture classification and effective robot arm control, enhancing the collaborative workflow of the system and reducing operational strain. This approach could further be used in medical applications, such as laparoscope manipulation in Minimally Onvasive Surgery (MIS).