An Augmented Reality Programming Approach to Incorporate Spatial Multimodal Interaction

Abstract

Abstract With the wide application of mobile robots, traditional robot programming faces a series of challenges due to its complex task settings and user interactions. In the context of Industry 5.0, we are committed to researching a spatial programming method based on AR (augmented reality) technology to realize human–robot collaboration. In this article, the AR-based spatial perception method is studied, and the virtual coordinate system in AR is established through the virtual reality model fusion perception method. By analyzing the relationship between each virtual coordinate system, the robot virtual space operation is planned, and then the physical world robot operation is controlled. A spatial multimodal AR interaction interface covering gesture, speech, EEG signals, and gaze control is designed to realize different functions for each modal interaction method. In the laboratory environment, the feasibility and effectiveness of the system are tested and verified by taking the coal mining machine interceptor tooth repair task as an example and selecting multiple coal mining machine interceptor tooth repairs for flexible operation tasks in the testing process. The results of the study show that the spatial programming method based on AR technology can significantly improve the efficiency of task execution of operator-controlled mobile robots and enhance the human–robot interaction experience.