PanoSCU: A Simulation-Based Dataset for Panoramic Indoor Scene Understanding

Abstract

Panoramic images play an important role in indoor robotics, offering a comprehensive spatial understanding of the environment. For tasks such as visual room rearrangement, where agents restore objects to their original positions or states, panoramic views provide a broader understanding of all changes. Existing 2D datasets for scene change understanding are limited to single-view images, which fail to capture the full spatial context, object relationships, and partially visible or obscured objects, making them less suitable for embodied AI applications. To address this, we introduce PanoSCU (Panoramic Scene Change Understanding), a dataset specifically designed to enhance the visual object rearrangement task. PanoSCU supports eight research tasks using single-viewand panoramic inputs: single-viewand panoramic detection, single-view and panoramic segmentation, single-view and panoramic change understanding, embodied object tracking, and change reversal. Our dataset contains 5,300 simulator-generated panoramas, encompassing 48 frequently-seen indoor object classes. We also introduce PanoStitch, a training-free stitching method for automatic panoramic data collection in embodied AI simulators. We evaluate state-of-the-art methods on panoramic segmentation and change understanding tasks. There is a gap in existing methods, as they are not designed for panoramic inputs and struggle with varying ratios and sizes, resulting from the unique challenges of visual object rearrangement. Our findings reveal these limitations and highlight PanoSCU’s challenging nature, emphasizing the need for further research to bridge this gap for panoramic indoor scene understanding. The dataset and the baseline methods for panoramic segmentation task are available publicly a.