A Multimodal Hand Movement Recognition Framework Based on S-Transform and ISDNet

Abstract

In the rehabilitation of hand movement disorders, multimodal signal-based hand movement recognition (HMR) plays a crucial role in enhancing therapeutic interventions and improving patient outcomes. However, existing methods face challenges such as suboptimal feature fusion and limited recognition performance. To address these issues, this article proposes a novel multimodal HMR framework. First, a signal fusion algorithm based on Spearman’s rank correlation coefficient (SRCC) is utilized to effectively integrate features from surface electromyography (sEMG) and triaxial acceleration signals (TASs), laying a solid foundation for subsequent feature fusion. Next, a feature fusion algorithm based on S-transform (S-T) and RGB image technology is developed, transforming signals into 3-D time-frequency fusion feature maps (3D-TFTTMs) to more comprehensively capture the time-frequency characteristics of the signals. Subsequently, a deep learning model, inception-SENet-DenseNet (ISDNet), is designed, incorporating both inception and squeeze-and-excitation network (SENet) modules. The inception module extracts fused features, while SENet dynamically adjusts channel weights, significantly enhancing recognition performance. Evaluation on the Ninapro DB2&3, DB5, and DB7 databases demonstrates that ISDNet achieves HMR accuracies of 97.02%, 93.78%, and 95.37%, respectively, significantly outperforming existing multimodal HMR methods. The results validate the effectiveness of the proposed framework in multimodal fusion and highlight its potential for advancing HMR technology, with broad application prospects in areas such as prosthetics, rehabilitation, and robotics.