Learning Hierarchical Control Structures for Multiple Tasks and Changing Environments

Abstract

While the need for hierarchies within control systems is apparent, it is also clear to many researchers that such hierarchies should be learned.Learning both the structure and the component behaviors is a di cult task.The bene t of learning the hierarchical structures of behaviors is that the decomposition of the control structure into smaller transportable chunks allows previously learned knowledge to be applied to new but related tasks.Presented in this paper are improvements to Nested Q-learning NQL that allow more realistic learning of control hierarchies in reinforcement e n vironments.Also presented is a simulation of a simple robot performing a series of related tasks that is used to compare both hierarchical and non-hierarchal learning techniques.0 5 Start Go to B 2 ,, Q 8 0 5 Go to door exit 3 , , , Q 1 0 5 Primitive action 3 , , , Q 3 0 4 3 , , , Q 3 0 3 3 , , , Q 0 1 3 3 , , , Q 7 1 4 Go to door entrance 4 , , , , Q 3 1 4 Primitive action 4 , , , ,