A State Space Filter for Reinforcement Learning in Partially Observable Markov Decision Processes

摘要

This paper presents a technique for reinforcement learning to deal with both discrete and continuous state space systems in POMDPs while keeping the state space of an agent compact. First, in our computational model for MDP environments, a concept of “ state space filtering ” has been introduced and constructed to make the state space of an agent smaller properly by referring to “ entropy ” calculated based on the state-action mapping. The model is extended to be applicable in POMDP environments by introducing the mechanism of utilizing effectively of history information. The extended model is capable of being dealt with a continuous state space as well as a discrete state space by the extended model. Here, the mechanism of adjusting the amount of history information is also introduced so that the state space of an agent should be compact. Moreover, some computational experiments with a robot navigation problem with a continuous state space have been carried out. The potential and the effectiveness of the extended approach have been confirmed through these experiments.