Fusion of depth, color, and thermal images towards digital twins and safe human interaction with a robot in an industrial environment

Abstract

Accurate detection of the human body and its limbs in real-time is one of the challenges toward human-robot collaboration (HRC) technology in the factory of the future. In this work, a new algorithm has been developed to fuse thermal, depth, and color information. Starting with the calibration of the sensors to each other, proceeding with data fusion and detection of human body parts, and ending with pose estimation. The proposed approach has been tested in various HRC scenarios. It has shown a significant decrease in errors and noise in industrial environments. Furthermore, it produces not only a higher positioning accuracy of the head and hands compared to the state of art algorithms (e.g., OpenPose), but it is also faster. Hence, such an algorithm could be joined with digital twins for safe and efficient human-robot collaboration.