HAMMER: Heterogeneous, Multi-Robot Semantic Gaussian Splatting

摘要

3D Gaussian Splatting offers expressive scene reconstruction and can model a broad range of visual, geometric, and semantic information. However, efficient real-time map reconstruction with data streamed from multiple robots and devices remains a challenge. To that end, we propose HAMMER, a server-based multi-robot Gaussian Splatting method that leverages ROS communication infrastructure to generate 3D, metric-semantic maps from asynchronous robot data-streams. HAMMER consists of (i) a one-time frame alignment module that transforms local SLAM poses and image data into a global frame and requires no prior relative pose knowledge, and (ii) an online module for continually training semantic 3DGS maps from streaming data. HAMMER handles mixed perception modes, adjusts automatically for variations in image pre-processing among different devices, and distills CLIP semantic codes into the 3D scene for language queries. In real-world experiments, HAMMER creates better maps compared to baselines and is useful for downstream tasks, such as semantic navigation (e.g., “go to the couch”). Accompanying content at hammer-project.github.io.