Synchronous Gesture Recognition and Arm Joint Angle Monitoring for Human‐Machine Interaction Using Multiple Flexible Ultrasonic Patches

Abstract

Abstract Wearable ultrasonic devices can non‐invasively monitor muscle activities for prosthetic hand control and human‐machine interaction (HMI). However, most existing ultrasonic probes for deep muscle tissue monitoring are designed with bulky structures, limiting their wearability for long‐term continuous monitoring. Additionally, current research primarily focuses on hand movement recognition, lacking simultaneous detection of movements in other upper limb joints. In this work, a strategy is proposed for simultaneous gesture recognition and upper limb angle measurements using multiple A‐mode ultrasonic patches, enabling real‐time control of a four‐degree‐of‐freedom (4‐DOF) virtual robotic arm. The patches are adhered to the skin using bio‐adhesive silicone gel, facilitating long‐term, continuous monitoring of muscle activities. By employing machine learning algorithms, 10 distinct hand gestures are accurately classified, achieving a recognition accuracy of 96%. Real‐time monitoring of the wrist, elbow, and shoulder joints is achieved by tracking variations in the corresponding muscle thickness, with maximum errors of 4° for the wrist, 5° for the elbow, and 10° for the shoulder. Gestures and joint angles are simultaneously monitored at a frequency of 40 Hz and wirelessly transmitted to real‐time control of a 4‐DOF virtual robotic arm.