A Methodology to Quantify Simulation- Versus-Reality Differences in Images for Autonomous Robots

Abstract

This contribution focuses on how the quality of synthetic images can be measured in the context of robotics and autonomous vehicle (AV) applications. We propose a validation methodology grounded in the sim-to-real difference exhibited by a perception algorithm that comes into play in the autonomy stack associated with a robot or AV of interest. By concentrating on the performance of the robot or AV perception algorithm, the methodology focuses on features that are relevant to the specific robot on a specific task. When handling images associated with complex environments, the methodology draws on semantic alignment to identify data suitable for comparison, thus eliminating the strenuous requirement that one should use paired real–synthetic datasets. The sim-to-real gap is measured statistically as the discrepancy between the performance distribution of the perception algorithm when it is presented with synthetic and real images. The approach yields both a gap metric and a content similarity measure, the latter derived from the proportion of real and synthetic data that can be suitably aligned. The proposed methodology is detailed and demonstrated in the context of two applications: 1) quantifying the realism of images produced in a laboratory scene, with the perception task performed by a widely adopted object detection algorithm and 2) assessing the realism of video-game-produced and machine-learning-enhanced images for many-class semantic segmentation in an urban environment.