A multimodal-signals-based gesture recognition method for human machine interaction

Abstract

As the capacity for machines to extend human capabilities continues to grow, the communication channels used also need to expand. Allowing machines to translate gestural command into robot movements can make the communication between humans and machines more similar to interpersonal communication. Yet for operating requirements, instantaneity and precision of the interaction must reach the application level in realistic scenarios, and the accuracy of results cannot be guaranteed by thejudgement of a single motion sensor. Therefore, exploring a gesture detection technology that integrates multisensor information is very necessary. The presented work takes a step towards real-time gesture detection by fusing multiple physiological signals with wearable motion sensors. An algorithm is presented for processing and extracting motion signal acquired via inertial measurement unit (IMU) and electromyography (EMG) with a high error-tolerant way of wearing, and it is applied to the gesture recognition model established in reasonable threshold and logical judgment. This enables real-time gesture detection of 24 various assembled movements such as rotating palms while bending arms, clenching fist when unwinding upper limb. The result of involving intentional behavior by analyzing electroencephalogram (EEG) is also added to the gesture recognition process for eliminating unconscious actions. Together, these pipelines offer efficient gesture vocabulary suitable for remotely controlling robots. Experiments evaluate classifier performance and interface efficacy. The system successfully detected 95.6% of 360 commands, and the average processing time was 25.4ms among all the 15 trials of the experiment.