Real-Time Gesture-Based Control of a Quadruped Robot Using a Stacked Convolutional Bi-Long Short-Term Memory (Bi-LSTM) Neural Network

Abstract

In recent years, advancements in robotics have driven a growing interest in enhancing human-robot interaction (HRI) for improved collaboration and effectiveness, particularly in critical scenarios like search and rescue (SAR) operations. This paper introduces an innovative approach for intuitive control of a quadruped robot, MiT Spot Robot, through hand gestures, using a Stacked Convolutional Bi-Long Short-Term Memory (Bi-LSTM) neural network model. To enable seamless and efficient human-robot interaction (HRI), this advanced model is integrated with the Robot Operating System (ROS) and the Gazebo simulation environment. We propose a robust hand gesture recognition system employing computer vision techniques that accurately interpret dynamic hand gestures in real time. The recognised gestures are mapped to specific locomotion and task commands, facilitating natural and intuitive control of the Quadruped Robot during search and rescue (SAR) operations. A comprehensive hyperparameter tuning approach using a grid search is implemented to optimise the model's performance. Our simulation-based experimentation in ROS/Gazebo validates the effectiveness and responsiveness of the proposed control scheme, showcasing its potential to enhance human-robot collaboration (HRC) in critical scenarios such as SAR missions.