Curriculum-guided skill learning for long-horizon robot manipulation tasks

Abstract

Robotic tasks often involve solving long-horizon problems. Seen under the reinforcement learning framework, the rewards provided in these problems are often sparse, which can be problematic for the learning process. In this context, dividing the long-horizon task into smaller ones represents a viable strategy to alleviate the credit assignment problem. Another approach generally used to help with this problem is curriculum learning. This paper combines both with a new skill-chaining learning algorithm that provides transition policies to bridge the gap between skills. Our approach begins by extracting meaningful skills from the states of an expert trajectory, using a heuristic method, which are subsequently used by the skill learning and the skill chaining algorithms. By leveraging the sequential order of the skills inside the demonstration, we propose a method to learn inter-skill transition policies to ensure the skills are appropriately chained. Our curriculum-based training approach enables an agent to learn action sequences that generalize inside a specific sub-task context. Using the information of a single demonstration, we show that our approach can solve a robotic manipulation task with similar performance to methods that rely on a large amount of data. Because our skill segmentation method detects which skills are present across demonstrations, we also show that our approach can reuse skills already learned in a zero-shot way.