An Evaluation of DDPG, TD3, SAC, and PPO: Deep Reinforcement Learning Algorithms for Controlling Continuous System

Abstract

Continuous systems are physical systems that can be stimulated by continuous and analog variables.The parameters or variables are within a range of values.An excellent continuous controlling policy enables the system to act appropriately and smoothly without much intervention, which can be useful in robotics, self-driving, industries, etc.The DRL algorithm has extensive applications in continuous systems control.This essay will explore the performance of four DRL algorithms, that is the Deep Deterministic Policy Gradient (DDPG), Twin Delayed DDPG (TD3), Soft Actor-Critic (SAC), and Proximal Policy Optimization (PPO) by using environment from the four of environments in Mujoco in Gym.Comparative experiments are done, and the highest rewards and the required number of iterations to converge are compared.The result of comparative experiments illustrates that these DRL algorithms can learn relatively appropriate policies in continuous controlling tasks.In particular, TD3 and SAC were found to be able to learn the controlling policy more effectively.Further research is needed to find better ways to adjust hyperparameters.