A State Space Filter for Reinforcement Learning in POMIDPs - Application to a Continuous State Space

Abstract

This paper presents a technique to deal with both discrete and continuous state space systems in POMDPs for reinforcement learning while keeping the state space of an agent compact. First, our computational model for MDP environments, where a concept of "state space filtering" has been introduced and constructed to make properly the state space of an agent smaller by referring to "entropy" calculated based on the state-action mapping, is extended to be applicable in POMDP environments by introducing the mechanism of utilizing effectively of history information. Then, it is possible to deal with a continuous state space as well as a discrete state space. In this, the mechanism of adjusting the amount of history information is also introduced so that the state space of an agent should be compact. Moreover, some computational experiments with a robot navigation problem with a continuous state space have been carried out. The potential and the effectiveness of the extended approach have been confirmed through these experiments