Towards Mutual-Cognitive Human-Robot Collaboration: A Zero-Shot Visual Reasoning Method

Abstract

Human-Robot Collaboration (HRC) is showing the potential of widespread application in today's human-centric smart manufacturing, as prescribed by Industry 5.0. To enable safe and efficient collaboration, numerous visual perception methods have been explored, which allows the robot to perceive surroundings and plan collision-free, reactive manipulations. However, current visual perception approaches can only convey basic information between robots and humans, falling short of semantic knowledge. With this limitation, HRC cannot guarantee smooth operation when confronted with similar yet unseen situations in real-world applications. Therefore, a mutual-cognitive HRC architecture is proposed to plan human and robot operations based on the learning of knowledge representation of onsite situations and task structures. A zero-shot visual reasoning approach is introduced to derive suitable teamwork strategies in the mutual-cognitive HRC from perceived results, including human actions and detected objects. It assigns adaptive robot path planning and knowledge support for humans by incorporating perception components into a knowledge graph, even when dealing with a new but similar HRC task. Lastly, the significance of the proposed mutual-cognitive HRC system is revealed through its evaluation in collaborative disassembly tasks of aging electric vehicle batteries.