Curriculum Learning Algorithms for Reward Weighting in Sparse Reward Robotic Manipulation Tasks

Abstract

Robotic learning from sparse rewards can be a considerable challenge due to large amounts of data required for mastering a task. We explore the application of curriculum learning (CL) algorithms for automatic reward weighting to tackle learning from sparse rewards in robotic pick-and-place and stacking tasks. We take several state-of-the-art CL algorithms that were originally designed to generate curriculum by manipulating the environment and appropriate them to weigh multiple sparse reward functions instead. The reward functions are chosen in a way that facilitates staged learning of the task, and the two robotic tasks are designed so that the agent learns to generalize to any initial and goal object position in the scene. The results of our three implemented CL algorithms show large improvement over the naive and state-of-the-art baselines in terms of speed of convergence to a successful policy in experiments with multiple task variations. Various generalization tests showcase some strengths and weaknesses of our approach. Inspection of changes in reward weight values during training further reveals varying curricula generated by the employed approaches, and showcases shifting emphasis from auxiliary to the main reward as the training progresses.