Hierarchical reinforcement learning with movement primitives

Abstract

Temporal abstraction and task decomposition drastically reduce the search space for planning and control, and are fundamental to making complex tasks amenable to learning. In the context of reinforcement learning, temporal abstractions are studied within the paradigm of hierarchical reinforcement learning. We propose a hierarchical reinforcement learning approach by applying our algorithm PI <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> to sequences of Dynamic Movement Primitives. For robots, this representation has some important advantages over discrete representations in terms of scalability and convergence speed. The parameters of the Dynamic Movement Primitives are learned simultaneously at different levels of temporal abstraction. The shape of a movement primitive is optimized w.r.t. the costs up to the next primitive in the sequence, and the subgoals between two movement primitives w.r.t. the costs up to the end of the entire movement primitive sequence. We implement our approach on an 11-DOF arm and hand, and evaluate it in a pick-and-place task in which the robot transports an object between different shelves in a cupboard.