Neural Network Dynamics for Model-Based Deep Reinforcement Learning with\n Model-Free Fine-Tuning

Abstract

Model-free deep reinforcement learning algorithms have been shown to be\ncapable of learning a wide range of robotic skills, but typically require a\nvery large number of samples to achieve good performance. Model-based\nalgorithms, in principle, can provide for much more efficient learning, but\nhave proven difficult to extend to expressive, high-capacity models such as\ndeep neural networks. In this work, we demonstrate that medium-sized neural\nnetwork models can in fact be combined with model predictive control (MPC) to\nachieve excellent sample complexity in a model-based reinforcement learning\nalgorithm, producing stable and plausible gaits to accomplish various complex\nlocomotion tasks. We also propose using deep neural network dynamics models to\ninitialize a model-free learner, in order to combine the sample efficiency of\nmodel-based approaches with the high task-specific performance of model-free\nmethods. We empirically demonstrate on MuJoCo locomotion tasks that our pure\nmodel-based approach trained on just random action data can follow arbitrary\ntrajectories with excellent sample efficiency, and that our hybrid algorithm\ncan accelerate model-free learning on high-speed benchmark tasks, achieving\nsample efficiency gains of 3-5x on swimmer, cheetah, hopper, and ant agents.\nVideos can be found at https://sites.google.com/view/mbmf\n