Implementation of a wireless gesture-controlled robotic arm utilizing MediaPipe

Abstract

In an age of advancing robotic technology, the current state of human-computer interface of robotic arms requires a level of expertise and training to utilize, especially within applications that involve kinematics. This paper develops a Human-Computer Interface (HCI) method to control a 3D printed articulated robotic arm, using Google’s MediaPipe, wirelessly via TCP communication. The hand tracking software developed can derive a user’s hand’s x, y and z position and hand rotation angle. The software is also able to detect whether a user’s hand is open or closed. The developed system translates the cartesian co-ordinates of the user’s hand into respective motor positions using geometrical inverse kinematic methods, processed by a Raspberry Pi model 4B. An ESP32 Wroom-32E microcontroller was used to control a series of servo motors and a stepper motor to position the robotic arm. The method used requires no additional hardware except for a PC laptop webcam hence making this a low-cost system.