Deep Learning Based Semantic Labelling of 3D Point Cloud in Visual SLAM

Abstract

Three-dimensional (3D) point cloud understanding is important for autonomous robots. However, point clouds are normally irregular and discrete. It is challenging to obtain semantic information from them. In this paper, we present a method to build a dense semantic map, which utilizes both two-dimensional (2D) image labels and 3D geometric information. The dense point cloud is built by using a state-of-the-art RGB-D SLAM system. It is further segmented into meaningful clusters using a graph-based method. Then, image keyframes during the SLAM process are used to extract semantic image labels by a convolution neural network (CNN). Finally, these semantic labels are projected to the point cloud clusters to achieve a 3D dense semantic map. The effectiveness of our method is validated on a popular public dataset.