Imaginary Hindsight Experience Replay: Curious Model-based Learning for\n Sparse Reward Tasks

Abstract

Model-based reinforcement learning is a promising learning strategy for\npractical robotic applications due to its improved data-efficiency versus\nmodel-free counterparts. However, current state-of-the-art model-based methods\nrely on shaped reward signals, which can be difficult to design and implement.\nTo remedy this, we propose a simple model-based method tailored for\nsparse-reward multi-goal tasks that foregoes the need for complicated reward\nengineering. This approach, termed Imaginary Hindsight Experience Replay,\nminimises real-world interactions by incorporating imaginary data into policy\nupdates. To improve exploration in the sparse-reward setting, the policy is\ntrained with standard Hindsight Experience Replay and endowed with\ncuriosity-based intrinsic rewards. Upon evaluation, this approach provides an\norder of magnitude increase in data-efficiency on average versus the\nstate-of-the-art model-free method in the benchmark OpenAI Gym Fetch Robotics\ntasks.\n