Enhanced continuous valued Q-learning for real autonomous robots

Abstract

Q-learning, a most widely used reinforcement learning method, normally needs well-defined quantized state and action spaces to obtain an optimal policy for accomplishing a given task. This makes it difficult to be applied to real robot tasks because of poor performance of learned behavior due to the failure of quantization of continuous state and action spaces. To deal with this problem, we proposed a continuous valued Q-learning (Takahashi et al., 1999) (hereafter, called CVQ-learning) for real robot applications. This method utilized a function approximation method for representing a action value function. In this paper, we point out that this type of learning method potentially has a discontinuity problem of optimal actions given a state. To resolve this problem, this paper proposes a method for estimating where discontinuity of optimal action takes place and for refining a state space for CVQ-learning. To show the validity of our method, we apply the method to a vision-guided mobile robot of which task is to chase the ball. Although the task is simple, the performance is quite impressive.