RGB-D Scene Classification: A Unified Framework with Vision Transformers and Contextual Models

Abstract

Indoor scene classification poses a significant challenge, crucial for advancements in robotics, augmented reality, and smart environments. This paper presents an innovative model designed for classifying indoor scenes using RGB-D images. The model seamlessly integrates unsupervised segmentation, Vision Transformers (VT), convolutional neural networks, and conditional random fields. The workflow starts with segmenting the RGB-D data through the DBSCAN algorithm, followed by feeding the segmented images into a Vision Transformer for feature extraction. The VT features two branches: one dedicated to capturing fine-grained local patch representations and another focused on modeling long-range dependencies through self-attention mechanisms. The resulting features are then processed by a ResNet-18 architecture for multi-object classification, enhancing the understanding of the scene's content. Ultimately, the classified objects and contextual information from the VT's relation modeling branch are merged within a Conditional Random Field framework for comprehensive scene classification. Testing on the NYU-Dv2 and SUN RGB-D datasets reveals that our approach significantly outperforms existing techniques in accurately classifying diverse indoor environments. This methodology sets the stage for enhanced scene understanding, with promising applications in robotics, augmented reality, and smart environments.