A Method of Fusing Gesture and Speech for Human-robot Interaction

Abstract

Aiming at the problem of using the unimodal model in human-robot interaction, in order to improve the operator's ability to control and interact with the robot and thus realize the natural and friendly interaction between human and robot, we propose a novel multimodal fusion architecture based on gesture and speech. Firstly, the convolutional neural network and Baidu API are used to recognize gesture and speech respectively. Secondly, the result of gesture prediction probability and the result of speech forward matching are normalized. Finally, the proposed multimodal fusion algorithm is used to fuse the two results, and the output results are filled into the intention slot. The operator's intention is determined by judging the fill integrity of intents the intention slot. The experimental results show that the proposed multimodal fusion architecture is superior to the unimodal model in recognition accuracy and interaction efficiency.