Design-Bench: Benchmarks for Data-Driven Offline Model-Based\n Optimization

Abstract

Black-box model-based optimization (MBO) problems, where the goal is to find\na design input that maximizes an unknown objective function, are ubiquitous in\na wide range of domains, such as the design of proteins, DNA sequences,\naircraft, and robots. Solving model-based optimization problems typically\nrequires actively querying the unknown objective function on design proposals,\nwhich means physically building the candidate molecule, aircraft, or robot,\ntesting it, and storing the result. This process can be expensive and time\nconsuming, and one might instead prefer to optimize for the best design using\nonly the data one already has. This setting -- called offline MBO -- poses\nsubstantial and different algorithmic challenges than more commonly studied\nonline techniques. A number of recent works have demonstrated success with\noffline MBO for high-dimensional optimization problems using high-capacity deep\nneural networks. However, the lack of standardized benchmarks in this emerging\nfield is making progress difficult to track. To address this, we present\nDesign-Bench, a benchmark for offline MBO with a unified evaluation protocol\nand reference implementations of recent methods. Our benchmark includes a suite\nof diverse and realistic tasks derived from real-world optimization problems in\nbiology, materials science, and robotics that present distinct challenges for\noffline MBO. Our benchmark and reference implementations are released at\ngithub.com/rail-berkeley/design-bench and\ngithub.com/rail-berkeley/design-baselines.\n