Using Markov-k Memory for Problems with Hidden-state

Abstract

TRACA (Temporal Reinforcement learning and Classification Architecture) is a new learning system developed for robot-navigation problems. One difficulty in this area is dealing with problems which contain hidden-state. TRACA solves hidden-state tasks by building Markov-k memory chains where k may be of arbitrary length. This seemingly simple strategy has a number of hidden complexities. These complexities include restricting the number of searches and internal structures created to within reasonable bounds and requiring different methods of assessment in retaining memory chains for different types of problems. These complexities and others are discussed along with the techniques implemented TRACA. TRACA’s techniques are evaluated on some difficult hidden-state problems which TRACA successfully solves while requiring less training trials than other state-of-the-art approaches.