Immersive Mixed Reality Framework for Enhanced Human-Robot Collaboration

Abstract

This paper focuses on developing a simulation platform that enhances Human-Robot Interaction through intuitive and safe interchanges between human operators and a UR5e robot by operating a Digital Twin via Mixed Reality. Using the ROS Noetic framework and Unity software for development, in conjunction with Microsoft HoloLens 2, this work presents a system based on Mixed Reality designed for collaborative robotic experiences. Moreover, a key innovation in this research is integrating a ZED camera for real-time human tracking, which enables the simultaneous simulation of humans and robots within a shared mixed environment. This advancement builds upon an existing digital twin of the robot. However, it also significantly enriches the system’s capabilities for ensuring safety and facilitating intuitive interactions, validating the implemented methodology through simulation and real-life experimental results. The paper examines the implications of this development in different industrial and research contexts. It highlights the growing significance of mixed-reality technologies in creating complex human-robot interaction systems that ensure safety and efficiency.