Learning Hand Gestures using Synergies in a Humanoid Robot

Abstract

Hand gestures are a natural way of communication and integrating them into robots could allow for more efficient human-robot collaboration. In recent years, researchers and roboticists have attempted to replicate human hand motor control using the concept of synergies. In this paper, we present a new approach to obtaining kinematic synergies from hand gestures using a single RGB camera. We capture real-time hand gestures using the MediaPipe framework and convert them to joint angular velocities. We then use dimensionality reduction to obtain kinematic synergies from the joint angular velocities. These synergies can be used to reconstruct new hand gestures. We translate these reconstructed hand movement patterns into a humanoid robot, Mitra. Our results show that it is possible to control most of the joints of the robot for performing hand gestures using only a few synergies. This is more efficient than other contemporary methods. Furthermore, robots and prosthetics that use synergy models could enable near-natural human-robot collaboration.