Probing AR-assisted seamless HRC assembly for industry 5.0: Multi-modal mutual cognition and LLM-driven knowledge reasoning

摘要

• Firstly, a multimodal fusion HRC scene perception method is proposed, leveraging Transformer-based models, the framework achieves real-time recognition of operator postures and actions, as well as lightweight detection of various components in the assembly scene, enabling a comprehensive understanding of the assembly environment. • Based on scene perception results, a human-centric mutual cognition safety framework for HRC is proposed. This system enables operators to access real-time information about the collaborative environment, prevents motion interference in shared spaces, reduces human-robot interaction risks, and fosters intelligent coexistence between operators and cobots. • Then, an LLM-driven assembly knowledge reasoning system is proposed that decomposes assembly tasks into planning sequences. This system integrates real-time assembly scene information with assembly craft workflows to construct a craft information model, forming an assembly expert knowledge system. It enables intelligent craft guidance, task recommendations, and solutions optimization through LLM assistance. • Finally, an AR-based HRC assembly assistance system is developed, which delivers enhanced visual guidance for assembly tasks. The system integrates the functionalities for multimodal mutual cognition and knowledge reasoning. Validation in practical HRC assembly scenarios demonstrates its effectiveness. The advancement of Industry 5.0 fosters human-centric manufacturing, aiming to enhance the well-being and needs of operators. Current research on Human-robot Collaboration (HRC) in the assembly scene is progressively evolving toward intelligent manufacturing with a human-centric focus, particularly in safety and interaction. However, in existing HRC assembly environments, the disjointed relationship between humans and robots presents challenges in handling complex manufacturing tasks. Cobots often struggle to accurately comprehend human actions and assembly contexts, while operators lack real-time insights into the current assembly scene. This mismatch increases the cognitive and physical burden on operators, ultimately reducing assembly efficiency and quality. To address this issue and achieve seamless HRC assembly, this paper proposes an Augmented Reality (AR) assisted HRC assembly method that integrates multimodal mutual cognition with Large Language Model (LLM) driven knowledge reasoning. Firstly, a Transformer-based multimodal fusion perception method for HRC scenarios is proposed to overcome the limitations of current cobots in understanding and responding to human commands and environmental changes in real-time. Based on this, a human-centric mutual cognition and safety framework for HRC is proposed to mitigate interaction risks and promote intelligent coexistence between operators and cobots. Secondly, an LLM-driven assembly knowledge reasoning system is proposed. By constructing an assembly craft information model with semantic associations and dynamic updating capabilities, the system facilitates decision support for HRC tasks and provides optimized assembly plan recommendations. This approach effectively leverages the respective advantages of human operators and cobots in executing assembly tasks. In addition, an AR-based HRC assembly assistance system is designed to enhance visual guidance during the assembly process while integrating the functionalities above. Finally, practical validation is conducted in real-world assembly tasks. Experimental results demonstrate that the proposed method offers efficient, intelligent, and visualized support for HRC assembly tasks, significantly reducing the cognitive load on operators while improving assembly efficiency and product consistency.