Benchmarking CAD search techniques

Abstract

While benchmark datasets have been proposed for testing computer vision and 3D shape retrieval algorithms, no such datasets have yet been put forward to assess the relevance of these techniques for engineering problems. This paper presents several distinctive benchmark datasets for evaluating techniques for automated classification and retrieval of CAD objects. These datasets include (1) a dataset of CAD primitives (such as those common in constructive solid geometry modeling); (2) two datasets consisting of classes generated by minor topological variation; (3) two datasets of industrial CAD models classified based on object function and manufacturing process, respectively; (4) and a dataset of LEGO© models from the Mindstorms© robotics kits. Each model in the datasets is available in three formats - ACIS SAT, ISO STEP, and as a VRML mesh (some models are available under several different fidelity settings). These are all available through the National Design Repository.Using these datasets, we present comprehensive empirical results for nińe (9) different shape and solid model matching and retrieval techniques. These experiments show, as expected, that the quality of precision-recall performance can significantly vary on different datasets. These experiments reveal that for certain object classes and classifications, such as those based on manufacturing processes, all existing techniques perform poorly. This study reveals the strengths and weaknesses of existing research in these areas, introduces open challenge problems, and provides meaningful datasets and metrics against which the success of current and future work can be measured.