A Decision Framework for AR, Dialogue and Eye Gaze to Enhance Human-Robot Collaboration

Abstract

Enabling an intuitive, bidirectional communication with real-time feedback to convey intentions and goals is essential in human-robot collaboration (HRC). In this paper, we propose ARDIE (Augmented Reality with Dialogue and Eye Gaze), a novel intelligent agent that leverages multi-modal feedback cues to enhance HRC. Our system employs a partially observable Markov decision process (POMDP) to formulate a joint decision policy integrating interactive augmented reality (AR), natural language, and eye gaze to provide real-time visual feedback to humans. Through object-specific AR renderings, ARDIE enables users to visualize current and future states of the environment, ultimately improving situational awareness and enhancing collaborative interactions between humans and robots.