An error-sensitive Q-learning approach for robot navigation

Abstract

Reinforcement learning can capture notions of optimal behavior occurring in natural systems. In the context of reinforcement learning, the learning rate controls how fast we modify our estimates. Generally Q-learning approach leverages the temporal-difference (TD) error to regulate Q-value, while utilizing a constant or decreasing learning rate, e.g., linear or polynomial learning rate, throughout the agent's life. Learning algorithm with polynomial learning rate learns faster at the cost of inferior trade-off between exploration and exploitation. None of them is evaluated based on the TD error. Whereas that cannot psychologically reflect the agent's true learning progress with unnecessary extra training episodes and exploration. This paper proposes an error-sensitive learning rate mechanism for Q-learning algorithm termed as (ESQL) to achieve better mitigation and faster learning. The agent is endowed sensibility to the TD error summed over the episodes. The derived method is implemented with indoor robot navigation task simulation in a stationary grid world environment. Experimental results are presented showing that ESQL approach achieves faster learning and latent better trade-off between exploration and exploitation compared with both constant and decreasing learning rate Q-learning approaches.