OmniDRL: An Energy-Efficient Deep Reinforcement Learning Processor With Dual-Mode Weight Compression and Sparse Weight Transposer

Abstract

In this article, we present an energy-efficient deep reinforcement learning (DRL) processor, OmniDRL, for DRL training on edge devices. Recently, the need for DRL training is growing due to the DRL’s distinct characteristics that can be adapted to each user. However, a massive amount of external and internal memory access limits the implementation of DRL training on resource-constrained platforms. OmniDRL proposes four key features that can reduce external memory access by compressing as much data as possible and can reduce internal memory access by directly processing compressed data. A group-sparse training (GST) enables a high weight compression ratio (CR) for every DRL iteration by selective utilization of weight grouping and weight pruning. A group-sparse training core is proposed to fully take advantage of compressed weight from GST by skipping redundant operations and reusing duplicated data. An exponent-mean-delta encoding additionally compresses the exponent of both weight and feature map for higher CR and low memory power consumption. A world-first on-chip sparse weight transposer enables the DRL training process of compressed weight without off-chip transposer. As a result, OmniDRL is fabricated in a 28-nm CMOS technology and occupies a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3.6\times 3.6$ </tex-math></inline-formula> mm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> die area. It shows a state-of-the-art peak performance of 4.18 TFLOPS and a peak energy efficiency of 29.3 TFLOPS/W. It achieved 7.42-TFLOPS/W energy efficiency for training robot agent (Mujoco Halfcheetah, TD3), which is 2.4 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times $ </tex-math></inline-formula> higher than the previous state of the art.