Real Time Computer Vision Based Robotic Arm Controller with ROS and Gazebo Simulation Environment

Abstract

Robotic arms are widely prevalent and find utility in a variety of applications. However, a significant and widespread challenge faced by these arms is their inability to replicate the intricate functionalities of a human hand, primarily due to the distinct structure of the hand. The primary objective of this work is to create a simulation of a robotic arm that can replicate the movements and functions of a human hand in real-time. Data obtained from hand and arm gestures created with Mediapipe will be transferred in real-time to the robotic arm that is visualized and simulated on ROS and Gazebo. Thus, the hand and arm movements of the user in front of the camera will be effective in real-time on the manipulable joints. This advancement holds the potential to facilitate the construction of a robot capable of emulating both the hand and the arm of humans with high fidelity, thereby enabling comprehensive control over the robotic arm’s actions in real-time.