Semantic map construction approach for human-robot collaborative manufacturing

Abstract

• A semantic map construction approach along with supported techniques is proposed, which integrates the semantic and geometric information of the human-robot collaborative manufacturing environment. • The features of dynamic objects are eliminated from the constructed semantic map to avoid the misuse of these features as the spatial reference to calculate the pose transformation between the two successive images, thus guaranteeing the accuracy of the robot's localization and pose estimation in the human-robot collaborative manufacturing environment. • The static and dynamic objects existing in the semantic map of the human-robot coexistence workspace can be differentiated by using the deep-learning based semantic segmentation, which offers the possibility to enable the human-like contextual reasoning and decision-making functionalities for robots thus achieving a more intensive human-robot collaboration in the future. Map construction is the initial step of mobile robots for their localization, navigation, and path planning in unknown environments. Considering the human-robot collaboration (HRC) scenarios in modern manufacturing, where the human workers’ capabilities are closely integrated with the efficiency and precision of robots in the same workspace, a map integrating geometric and semantic information is considered as the technical foundation for intelligent interactions between human workers and robots, such as motion planning, reasoning, and context-aware decision-making. Although different map construction methods have been proposed for mobile robots’ perception in the working environment, it is still a challenging task when applied to such human-robot collaborative manufacturing scenarios to achieve the afore-mentioned intelligent interactions between human workers and robots due to the poor integration of semantic information in the constructed map. On the one hand, due to the lack of ability for differentiating the dynamic objects, the mobile robot might sometimes wrongly use the dynamic objects as the spatial references to calculate the pose transformation between the two successive frames, which negatively affects the accuracy of the robot's localization and pose estimation. On the other hand, the map that integrates both the geometric and semantic information can hardly be constructed in real-time, which cannot provide an effective support for the real-time reasoning and decision making during the human-robot collaboration process. This study proposes a novel map construction approach containing semantic information generation, geometric information generation, and semantic & geometric information fusion modules, which enables the integration of the semantic and geometric information in the constructed map. First, the semantic information generation module analyzes the captured images of the dynamic working environment, eliminates the features of dynamic objects, and generates the semantic information of the static objects. Meanwhile, the geometric information generation module is adopted to generate the accurate geometric information of the robot's motion plane by using the environment data. Finally, a map integrating semantic and geometric information in real-time can be constructed by the semantic & geometric fusion module. The experimental results demonstrate the effectiveness of the proposed semantic map construction approach.