Domain Adaptation Using Vision Transformers and XAI for Fully Synthetic Industrial Training

Abstract

Deep learning algorithms are increasingly used to automate complex production tasks such as robot-based material handling and quality inspection. However, achieving high accuracy in proprietary industrial applications often requires extensive annotated datasets. To address this challenge, we propose a Domain Adaptation approach that integrates eXplainable AI (XAI) with feature extraction and dimensionality reduction techniques to identify key features and assess the sim-to-real gap using both qualitative and quantitative methods, without requiring annotated real data. After reducing the feature space via an XAI-based ablation study, we employ the self-supervised DINOv2 model with a Vision Transformer (ViT) backbone for foundational feature extraction. We then compare Principal Component Analysis (PCA) and Hierarchical Nearest Neighbor Embedding (HNNE) for dimensionality reduction and qualitative evaluation of sim-to-real affinity. To quantify this visual assessment, we explore the use of cosine distances between real and synthetic datasets in the low-dimensional feature space as a structured metric for sim-to-real alignment. We validate our approach on a proprietary automotive dataset and a public robotics dataset, both designed for sim-to-real benchmarking. Our method surpasses domain randomization on the automotive dataset, improving mAP@50-95 by 3.4%, and outperforms the state of the art on the public robotics dataset, achieving a 7.9% mAP@50 improvement. These results advance the state of the art in sim-to-real transfer and data rendering by enabling sim-to-real gap estimation without training. Finally, we propose steps to further improve our approach.