Applying Burn-in Strategy to Deep Reinforcement Learning with Spiking Neural Networks

Abstract

In recent years, neural networks in deep reinforce-ment learning (D RL) have achieved success in various tasks, owing to their high expressiveness and flexibility. Neural networks are expected to be increasingly used for robot control. However, large neural networks consume a significant amount of power during training and operation. Given that the power supply that can be installed in robots is limited, challenges remain in the practical application of robot control using DRL, which includes large-scale neural networks. Conversely, spiking neural networks (SNNs) have high power efficiency, which is achieved through dedicated computational elements. Previous studies reported that SNNs can be applied to DRL to learn a robot's walking motion in a virtual environment and reduce energy consumption. Although previous studies have applied SNNs to DRL, they have the drawback of recursive computation, causing output delays. In this study, we propose a method that combines SNNs with a structure that addresses the issues of previous studies using a twin-delayed deep deterministic policy gradient (TD3). In addition, a burn-in strategy was introduced to estimate the action values using the internal states of the SNNs. Experiments in a virtual environment showed that the proposed method achieved higher rewards than the previous methods.