AugRE: Augmented Robot Environment to Facilitate Human-Robot Teaming and Communication

Abstract

Augmented Reality (AR) provides a method to superimpose real-time information on the physical world. AR is well-suited for complex robotic systems to help users understand robot behavior, status, and intent. This paper presents an AR system, Augmented Robot Environment (AugRE), that combines ROS-based robotic systems with Microsoft HoloLens 2 AR headsets to form a scalable multi-agent human-robot teaming system for indoor and outdoor exploration. The system allows multiple users to simultaneously localize, supervise, and receive labeled images from robotic clients. An overview of AugRE and details of the novel system architecture that allows for large-scale human-robot teaming is presented below. Studies showcasing system performance with multiple robotic clients are presented. Results show that AugRE can scale to 50 robotic clients with minimal performance degradation, due in part to key components that leverage a recent advancement in robotic client-to-client communication called Robofleet. Finally we discuss new capabilities enabled by AugRE.