Multimodal Gesture Recognition for Mascot Robot System Based on Choquet Integral Using Camera and 3D Accelerometers Fusion

Abstract

A multimodal gesture recognition method for mascot robot system is proposed based on Choquet integral by fusing camera and 3D accelerometer data. The optimization of two fuzzy measures in the training phase for two recognition units, i.e., camera-based and accelerometer-based units, obtains enough recognition rate of 92.7% for 8 types of gestures by improving the recognition rate approximate 20% compared with that of each unit. The proposed method targets casual communication from humans to robots by integrating nonverbal gesture messages and verbal messages.