Deep Q-Learning (DQN)

Abstract

This chapter takes a deep dive into Q-learning combined with function approximation using neural networks. Q-learning in the context of deep learning using neural networks is also known as Deep Q Networks (DQN). The chapter first summarizes what I have talked about so far with respect to Q-learning. You will then look at code implementations of DQN on simple problems. Following this, you will take stock of Gymnasium and learn how it differs from OpenAI Gym. The next focus is on the Stable Baselines 3 (SB3) library and its associated ecosystem. You will also explore hyperparameter optimization using Optuna and plotting, both in the context of SB3. With the basic machinery in place, I will extend DQN to cover Atari game playing agent. The topics of DQN are followed by a quick exploration of various other RL environments and associated libraries covering the use of RL in financial market trading and in the field of robotics. However, be aware that these are still research environments useful to gain more insights. As some of these have multi-dimensional continuous value actions, there are better algorithms than DQN for training agents. This chapter focuses on learning about these environments and how to set them up. You will apply DQN where appropriate and detailed coverage of those algorithms appears in future chapters.