Deep Learning based Image Reconstruction from Brain Data during Object-Grasping Task with a BMI

Abstract

Brain-machine interface systems (BMI) have allowed the control of prosthetics and robotic arms using brainwaves alone to do simple tasks such as grasping an object. However, the low throughput information of brain-data decoding does not allow the robotic arm to achieve diverse grasp configurations for different types of objects. On the other hand, computer vision researchers have mostly solved the problem of robot hand configuration for object-grasping given visual object recognition. It is then natural to think that if the robotic arm could decode from brain data the image of the object that the user intends to grasp, then it could automatically decide the type of grasping to execute. For this reason, in this paper, we propose a method to decode visual representations of objects from brain data towards improving robot arm grasp configurations. More specifically, we recorded EEG data during an object-grasping experiment in which the participant had to control a robotic arm using a BMI to grasp an object. We also recorded images of the object and developed a multimodal representation of the encoded brain data and object image. Given this representation, the objective was to reconstruct the image given that only half of the image (the brain data encoding) was provided. To achieve this goal, we developed a deep convolutional autoencoder that learned a noise-free joint manifold of brain data encoding and the object image. After training, the autoencoder was able to reconstruct the missing part of the object image given that only brain data encoding was provided.