Vision-Based Robotic Manipulation of Intelligent Wheelchair with Human-Computer Shared Control

Abstract

Based on human-computer shared control, this paper introduces a novel robotic manipulation fashion combining computer vision and brain-computer interface (BCI). Designed for disabled groups, the intelligent wheelchair with our proposed method exhibits the precise robotic manipulation ability but also the human decision-making capabilities, which will bring better life quality for the disabled. The overall pipeline includes three parts: asynchronous brain-computer interface based on steady-state visual evoked potential (SSVEP), vision detection with deep network and robotic manipulation of UR5 robot. Particularly, first, the user receives the periodic visual stimulation with different frequencies and then electroencephalography (EEG) signals of the user are collected by EEG cap. Second, we preprocess the EEG signals and extract feature embedding. To judge the frequency of the stimulus signals received by the user, the canonical correlation analysis (CCA) algorithm is used to fit and compare it with the standard EEG signal. In our work, the signals with different frequencies corresponds to different types of objects item by item. Third, we apply the off-the-shelfvision detection algorithm, Mask-RCNN, to output the position of the object corresponding to the detected EEG in the image frame. UR5 robot arm plan manipulation path according to the position of objects transferred by robot operating system (ROS). Extensive experiments show that our method can achieves performance with more than 90% accuracy and the user can control the robot arm to grab the expected object accurately through BCI.