Hindsight Experience Replay With Experience Ranking

Abstract

Reinforcement Learning (RL) algorithms face difficulties when dealing with robotic tasks in sparse reward settings and as a result, they often require millions of interactions with the environment to learn successfully. A recent algorithm Hindsight Experience Replay (HER) was introduced to tackle this difficulty by adding virtual goals and therefore increase significantly the sample-efficiency by learning in transitions when the robot does not achieve the original goal. However, these additional goals are sampled randomly from each episode batch of transitions, which might have no relationship with the original goal. This might make learning with the original goal slower due to the bad influence of irrelevant virtual goals. In this paper, we address this issue by applying experience ranking (ER) to these additional goals. We first compare each sampled virtual goal and the original goal and then compare the difference with a threshold. Transitions in which the robot achieves a virtual goal that is not close to the original goal are filtered out, and the remaining are used for training the policy. The improvement in learning performance is validated in four simulated robotic tasks. The experiment results show significant improvement in terms of the learning speed and robustness.