Consolidated Adaptive T-soft Update for Deep Reinforcement Learning

Abstract

Demand for deep reinforcement learning (DRL) is gradually increased to enable robots to perform complex tasks, while DRL is known to be unstable. As a technique to stabilize its learning, a target network that slowly and asymptotically matches a main network is widely employed to generate stable pseudo-supervised signals. Recently, T-soft update has been proposed as a noise-robust update rule for the target network and has contributed to improving the DRL performance. However, the noise robustness of T-soft update is specified by a hyperparameter, which should be tuned for each task, and suppression of updates would cause deviation of the two networks. This study develops a consolidated and adaptive T-soft (CAT-soft) update based on approximate maximum likelihood estimation of student-t distribution and an additional consolidation. Since the noise robustness is represented by a model parameter of the student-t distribution, this method makes the noise robustness adaptive. In addition, the parameters of the main network, those that deviate from the target network, are consolidated to the target network. The proposed method outperformed the conventional methods in numerical simulations.