Onboard Semantic Mapping for Action Graph Estimation

Abstract

Task planning is a pivotal component of autonomous mobile robots, requiring them to formulate a sequence of actions to achieve a predefined final goal. This requires a semantic understanding of the scene, enabling the robot to comprehend its surroundings and identify objects available for interaction. While some systems rely on environmental sensing for accurate estimation of the obstacles, they can only achieve this under controlled environments and is not flexible nor scalable. In this paper, we address these problems by introducing a new concept of onboard semantic mapping for action graphs. We create an environment map relying solely on an onboard RBG-D camera sensor and generate a graph representing all action sequences that the robot can perform over time. In our method, we integrate a 3D object detector with a visual SLAM pipeline to construct a semantic map, which contains only essential information for the action graph generation. The map is systematically fed into the action graph generator, which computes the potential actions of the robot and the subsequent states of the environment. We evaluate our approach on real-world scenarios in a qualitative and quantitative manner.