A Multimodal Construction Method for a Digital Twin System with Human-Robot Collaboration

Abstract

Manufacturing is shifting from traditional machine-centric systems to human-centric ones. Digital twin technology is a crucial link enabling the digital and intelligent transformation of manufacturing. However, current digital twins focus only on representing manufacturing resources and lack the capability to characterize complex human–robot collaborative scenarios. To address this, we propose a multimodal sensing method for constructing digital twins of human–robot collaboration scenarios. Specifically, we use different modeling techniques for physical space elements with different characteristics. Manufacturing resources are encapsulated and digitally modeled in a hierarchical manner. Humans are mapped in real time using neural networks. Additionally, we improve human mesh recovery algorithms for occluded human body. Experiments demonstrate our method outperforms baseline approaches. Finally, we integrate humans and robots in a shared space, achieving digital twins of human– robot hybrid scenarios.