Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map\n Them to Actions

Abstract

We transform reinforcement learning (RL) into a form of supervised learning\n(SL) by turning traditional RL on its head, calling this Upside Down RL (UDRL).\nStandard RL predicts rewards, while UDRL instead uses rewards as task-defining\ninputs, together with representations of time horizons and other computable\nfunctions of historic and desired future data. UDRL learns to interpret these\ninput observations as commands, mapping them to actions (or action\nprobabilities) through SL on past (possibly accidental) experience. UDRL\ngeneralizes to achieve high rewards or other goals, through input commands such\nas: get lots of reward within at most so much time! A separate paper [63] on\nfirst experiments with UDRL shows that even a pilot version of UDRL can\noutperform traditional baseline algorithms on certain challenging RL problems.\nWe also also conceptually simplify an approach [60] for teaching a robot to\nimitate humans. First videotape humans imitating the robot's current behaviors,\nthen let the robot learn through SL to map the videos (as input commands) to\nthese behaviors, then let it generalize and imitate videos of humans executing\npreviously unknown behavior. This Imitate-Imitator concept may actually explain\nwhy biological evolution has resulted in parents who imitate the babbling of\ntheir babies.\n