SIRL: Self-Imitation Reinforcement Learning for Single-step Hitting Tasks

Abstract

Reinforcement learning (RL) has demonstrated significant success in various sequential decision-making tasks. However, standard RL frameworks suffer from low efficiency in single-step robotic hitting tasks, which require accurate control and single-step decision-making under delayed reward. To address this challenge of single-step tasks, a Self-Imitation Reinforcement Learning (SIRL) algorithm is proposed to better utilize each interaction sample. With SIRL, the agent can obtain optimal successful samples during itself interactions without human demonstrations, even if the actual interaction fails. The proposed SIRL uses the self-imitation learning of these optimal samples to accelerate the learning of RL policy. In this paper, we create two challenging hitting tasks in MuJoCo simulation, Slide, and TableTennis, to evaluate our approach. Experimental results demonstrate that the proposed SIRL algorithm outperforms the standard RL methods and supervised learning methods in terms of both sample efficiency and performance. Especially, in sparse reward settings, SIRL stands out as the only RL-based method that can learn these tasks, as self-imitation learning provides the agent with more gradient information for policy optimization.