Seamless Robot Teleoperation: Intuitive Control through Hand Gestures and Neural Network Decoding

Abstract

Robotic teleoperation has enabled remote interaction with hazardous environments, overcoming spatial constraints on human perception and manipulation. Most teleoperation systems rely on task-dependent interfaces to generate human instructions. This can lead to barriers in familiarizing the robot’s workspace and thus increase the training time for less experienced users. In order to address these problems, we introduce a novel hand gestures based robot teleoperation method, eliminating the need for specialized controlling devices. Leveraging hand landmark detection and a neural network-based decoding algorithm, the system interprets hand movements to control robot velocity, offering a user-friendly solution to communicating with the robot. Our trained model achieves an F2 score of 0.994 and outperforms algorithms in the collected dataset. Furthermore, the proposed method has been validated on a real-world Franka robot, achieving success rates of 100%, 80%, and 86.7% across three manipulation tasks.