Empirical study of deep reinforcement learning algorithms for CartPole problem

Abstract

CartPole problem is a classical control problem, which is important in many fields, including robotics and self-driving cars. Deep reinforcement learning is applied for controlling the CartPole balancing problem. This study compares the performance of four deep reinforcement learning algorithms: Deep Q-Network (DQN), Proximal Policy Optimization (PPO), QR (Quantile Regression) DQN, and Advantage Actor-Critic (A2C). The CartPole balance problem in the OpenAIGym environment is taken into consideration to execute the deep reinforcement learning algorithms. Many deep Reinforcement algorithms exist, and selecting the right one might be difficult depending on the problem’s characteristics. Our findings can help in the ranking of the four methods based on how well they fit the requirements of a given application. The performance in terms of training time, training timesteps, maximum value of episode-reward-mean, and average frames per second for each method are evaluated. The results demonstrate that the best approach for resolving this problem is to use PPO method with a set of prespecified hyperparameters.