A hybrid architecture for hierarchical reinforcement learning

摘要

Autonomous robot systems operating in the real world have to be able to learn new tasks and environmental conditions without the need for an outside teacher. While reinforcement learning represents a good formalism to achieve this, its long learning times and need for extensive exploration often make it impracticable for online learning on complex systems. The hybrid architecture presented in this paper addresses this issue by applying reinforcement learning on top of an automatically derived abstract discrete event dynamic system (DEDS) supervisor. This reduces the problem of policy acquisition within this approach to learning to coordinate a set of closed-loop control strategies in order to perform a given task. Besides dramatically reducing the complexity of the learning task this framework also permits the incorporation of a priori knowledge and facilitates the inclusion of learned policies as actions in order to transfer skills to new task domains. To demonstrate the applicability of this approach, the architecture is used to learn locomotion gaits on a four-legged robot platform.