ToolNavigator: Dataset Generation for Small Tools Handling and Vision-Language Navigation in Construction Sites via Simulation for Robots

Abstract

Construction sites present complex and dynamic environments where small tools contribute to a significant proportion of accidents and hazards. There doesn’t exist any datasets for vision-language navigation (VLN) in construction sites, restricting the development of AI models for autonomous navigation and tool handling. This paper introduces ToolNavi-gator, a simulation-driven dataset for small tool handling and VLN at construction sites. By integrating Blender with NVIDIA Isaac Sim, we create a scalable and customizable dataset that replicates real-world construction scenarios. ToolNavigator contains diverse construction site configurations, 543 small tools, 102 types of equipment, and 22 categories of heavy machinery. The dataset includes rich multimodal annotations such as 2D/3D bounding boxes, depth maps, semantic masks, and scene graphs to support enhanced spatial reasoning and object interaction. Experimental results demonstrate the usefulness of ToolNavigator for creating challenging navigation tasks for tools in construction sites. Previous state-of-the-art models trained on ToolNavigator did not achieve high performance in navigation tasks, with Maplm achieving only a 48.9% success rate in 5-shot learning and 53.2% in full training for unseen environments. These results highlight ToolNavigator’s potential to advance AI-driven automation in construction sites.