Multimodal Computer Vision Framework for Human Assistive Robotics

Abstract

This paper presents a multimodal computer vision framework for human assistive robotics with the purpose of giving accessibility to persons with disabilities. The user is capable of interacting with the system just by staring. Specifically, it is possible to select the desired object as well as to indicate the intention to grasp it just by staring at it. This gaze information is provided by ©Tobii Glasses 2 that in combination with a deep learning algorithm gives the class id of the desirable object. Later, the object's pose is estimated using a RGB-D camera with a new developed technique. This technique mixes a template based algorithm with a deep learning algorithm giving a precise, realtime method for pose estimation. Once the pose is obtained, it is transformed to a grasping position in the coordinate system of the assistive robot that performs the grasping operation.