UnRealTHASC – A Cyber-Physical XR Testbed for Underwater Real-Time Human Autonomous Systems Collaboration

Abstract

Research on underwater human-robot teaming is particularly promising because of complementary sensory skills that may overcome unique challenges associated with the combination of limited connectivity and low visibility. Nevertheless, testing human-robot collaboration under water, especially in complex, real-world scenarios, poses severe safety, cost, and time constraints that significantly hinder academic research in this space. This paper presents a novel cyber-physical extended-reality (XR) testbed, referred to as Underwater Real-Time Human Autonomous Systems Collaboration (UnRealTHASC), designed to enable human-robot interactions in simulated photorealistic environments with mixed real and virtual wearables, and advanced autonomous underwater sensors in-the-loop. Novel sensor interfaces are designed to integrate real and virtual sensors for measuring physiological and cognitive human states underpinning decision-making abilities. Physics-based human and robot motion models are developed along with new sensor simulations in order to capture the couplings between underwater behaviors and perception based on measurements from optical and sonar sensors. Real-time data acquisition pipelines are created to access and share data from both real and virtual sensors and robots, such that new methods for online planning and collaboration may be tested via human-in-the-loop demonstrations.