Benchmarks for Cloud Robotics

Abstract

Several areas of computer science, including computer vision and naturallanguage processing, have witnessed rapid advances in performance, due in partto shared datasets and benchmarks. In a robotics setting, benchmarking ischallenging due to the amount of variation in common applications: researchers can usedifferent robots, different objects, different algorithms, different tasks, anddifferent environments.Cloud robotics, in which a robot accesses computation and data over a network,may help address the challenge of benchmarking in robotics. By standardizingthe interfaces in which robotic systems access and store data, we can define acommon set of tasks and compare the performance of various systems.In this dissertation, we examine two problem settings that are well served bycloud robotics. We also discuss two datasets that facilitate benchmarking ofseveral problems in robotics. Finally, we discuss a framework for defining andusing cloud-based robotic services.The first problem setting is object instance recognition. We present aninstance recognition system which uses a library of high-fidelity object modelsof textured household objects. The system can handle occlusions, illuminationchanges, multiple objects, and multiple instances of the same object.The next problem setting is clothing recognition and manipulation. We propose amethod that enables a general purpose robot to bring clothing articles into adesired configuration from an unknown initial configuration. Our method uses alibrary of simple clothing models and requires limited perceptual capabilities.Next, we present BigBIRD (Big Berkeley Instance Recognition Dataset), which hasbeen used in several areas relevant to cloud robotics, including instancerecognition, grasping and manipulation, and 3D model reconstruction. BigBIRDprovides 600 3D point clouds and 600 high-resolution (12 MP) images coveringall views of each object, along with generated meshes for ease of use. We alsoexplain the details of our calibration procedure and data collection system,which collects all required data for a single object in under five minutes withminimal human effort.We then discuss the Yale-CMU-Berkeley (YCB) Object and Model Set, which isspecifically designed for benchmarking in manipulation research. For a set ofeveryday objects, the dataset provides the same data as BigBIRD, an additionalset of high-quality models, and formats for use with common robotics softwarepackages. Researchers can also obtain a physical set of the objects, enablingboth simulation-based and robotic experiments.Lastly, we discuss Brass, a preliminary framework for providing robotics andautomation algorithms as easy-to-use cloud services. Brass can yield severalbenefits to algorithm developers and end-users, including automaticresource provisioning and load balancing, benchmarking, and collective robotlearning.