Improving Robot Proficiency Self-Assessment Via Meta-Assessment

摘要

Proficiency self-assessment (PSA), which is the ability to estimate how likely one can complete a task, is a beneficial property for autonomous robots. Prior work developed the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">assumption-alignment tracking</i> (AAT) method for PSA, which estimates the probability that a robot will successfully complete a task. This paper refers to the prediction made by AAT as the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">first-level assessment</i> (FLA), and further proposes a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">second-level assessment</i> (SLA) that determines whether the FLA prediction is correct. The probability that the FLA prediction is correct is conditioned on four <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">features:</i> (1) the mean distance from a test sample to its nearest neighbors in the training set; (2) the predicted probability of success made by the FLA; (3) the ratio between the robot's current performance and its performance standard; and (4) the percentage of the task the robot has already completed. The SLA model is trained on the four features using a Random Forest algorithm. It is evaluated by two metrics: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">discriminability</i> , measured by the area under the ROC curve, and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">calibration</i> , measured using expected calibration error. On a simulated navigation task and a manipulation task by a Sawyer robot, results demonstrate that the SLA model not only calibrates the FLA model as well as existing calibration methods (Platt calibration and isotonic regression), but also produces very high discriminability even if the FLA model's original discriminability is much lower. Results also indicate the usefulness of each of the four features used by the SLA model.