What You See is what You Grasp: User-Friendly Grasping Guided by Near-eye-tracking

Abstract

This study introduces an advanced human-robot interface designed to discern and execute manipulation tasks based solely on visual cues. The interface combines eye-tracking technology and robotic manipulation, facilitating actions like grasping or pick-and-place tasks. We have developed a head-mounted device for tracking eye movements, allowing the system to determine the user's focus and initiate sight-driven manipulation. Enhancing grasping efficiency, the system incorporates a transformer-based model, utilizing attention blocks for feature extraction and optimizing both channel capacity and spatial resolution of the feature maps. Our experiments confirm the system's capability in aiding users to perform tasks using only their gaze, suggesting significant implications for assistive robots in helping people with upper limb disabilities or the elderly with everyday activities.