Gaussian Process Model based Reinforcement Learning

Abstract

Reinforcement learning (RL) has been a promising approach in robotics and control because data-driven learning methods can reduce system reliance on human engineering knowledge. A model-based RL autonomously learns observed dynamics based on a general flexible nonparametric approach. Probabilistic Inference for Learning COntrol (PILCO) is one of the most data-efficient model-based RL frameworks. Since PILCO sets up a Bayesian estimator problem with a Gaussian process regression, it derives a fully deterministic approximate inference for policy evaluation, which makes it computationally efficient. However, PILCO requires a task-specific scenario. If an agent is given a new goal that is different than the original training goal, PILCO should relearn its model from scratch. This paper extends PILCO to tune a linear feedback controller with a quadratic cost function, where the quadratic cost function commonly used in control systems can adjust the trade-off relationship between control input consumption and convergence rate. The suggested method is not only able to maintain the analytic and deterministic approximate inference for policy evaluation, but is also able to interpret the controller design. The suggested RL framework is applied to the control of a small quadrotor unmanned aerial vehicle (UAV) with no given dynamics. The simulation results show the convergence of the learning control performance as a function of the number of RL iterations.