Sample Complexity of Estimating the Policy Gradient for Nearly\n Deterministic Dynamical Systems

Abstract

Reinforcement learning is a promising approach to learning robotics\ncontrollers. It has recently been shown that algorithms based on\nfinite-difference estimates of the policy gradient are competitive with\nalgorithms based on the policy gradient theorem. We propose a theoretical\nframework for understanding this phenomenon. Our key insight is that many\ndynamical systems (especially those of interest in robotics control tasks) are\nnearly deterministic -- i.e., they can be modeled as a deterministic system\nwith a small stochastic perturbation. We show that for such systems,\nfinite-difference estimates of the policy gradient can have substantially lower\nvariance than estimates based on the policy gradient theorem. Finally, we\nempirically evaluate our insights in an experiment on the inverted pendulum.\n