Indoor Scene Classification Using RGB-D Data: A Vision Transformer and Conditional Random Field Approach

Abstract

The classification of indoor scenes using RGBD images poses some challenges due to the nature of indoor environments and the vast variety of objects that may be in an environment. However, it can significantly improve the methods in robotics, augmented reality and smart environments. In this study, we introduce a new model for indoor scene classification, leveraging unsupervised segmentation, vision transformers, convolutional neural networks, and conditional random fields. The process starts with the segmentation of RGB-D data by DB-GMM that offers useful image segments. These segments are then passed into a Vision Transformer (VT) designed for robust feature extraction. The VT model consists of two main components: a feature extraction branch, capturing intricate local details within each patch, and a relational branch that builds long-range dependencies using selfattention mechanisms. This dual approach enables the VT to capture both the local and context features which are fed to a ResNet-18 model for multi-object recognition for rich scene composition information. To improve the final scene classification, object labels and the relational features are collected in the VT are fed into a Conditional Random Field (CRF), which offers a more structured representation of the scene. We evaluated our method on the NYU Depth V2 and SUN RGB-D datasets, demonstrating strong accuracy in distinguishing varied indoor scenes, surpassing many current methods.