VITRO - Model based vision testing for robustness

Abstract

This paper introduces a model-based approach for testing robustness of computer vision solutions with respect to a given task or application. Assessment of essential CV component robustness is crucial to ensure a safe robot and human coexistence. Currently this is mostly a manual and heuristic task lacking reliable metrics for determining the completeness and strength of a given test set. Our novel approach enables the generation of test data with a measurable coverage of optical situations both typical and critical for a given application. Typical situations are defined using a specific domain model while critical circumstances can be selected from a list of predefined hazards which was created using a proven hazard analysis procedure. Furthermore, the framework allows the automatic reduction of redundancy over the entire set of test images by using clustering. Finally the required oracle (ground truth) is automatically generated and is correct by definition.