Inverse reinforcement learning from failure

Abstract

<em>Inverse reinforcement learning</em> (IRL) allows autonomous agents to learn to solve complex tasks from successful demonstrations. However, in many settings, e.g., when a human learns the task by trial and error, <em>failed</em> demonstrations are also readily available. In addition, in some tasks, purposely generating failed demonstrations may be easier than generating successful ones. Since existing IRL methods cannot make use of failed demonstrations, in this paper we propose <em>inverse reinforcement learning from failure</em> (IRLF) which exploits both successful and failed demonstrations. Starting from the state-of-the-art <em>maximum causal entropy IRL</em> method, we propose a new constrained optimisation formulation that accommodates both types of demonstrations while remaining convex. We then derive update rules for learning reward functions and policies. Experiments on both simulated and real-robot data demonstrate that IRLF converges faster and generalises better than maximum causal entropy IRL, especially when few successful demonstrations are available.