Stability Guaranteed Actor-Critic Learning for Robots in Continuous Time

摘要

Actor-Critic (AC) architecture has the salient feature, for the plethora of Reinforcement Learning schemes, that two intertwining neural networks (NN) collaborate to deploy a motor learning mechanism that oversees and evaluates the action to control a dynamical system as the robot manipulator. Since such NNs can be studied as dynamical systems to learn how to control the robot dynamics with a given performance, it has finally paved the way to deal with stability analysis, unfortunately, few works have addressed it. In this paper, we propose a Critic-NN whose approximation of the value function substantiates the decision making mechanism that collaborates to tune the Actor-NN action (approximation of inverse dynamics). The novel proposed design of the adaptation of the neural weights yields Lyapunov stability that provides explicit conditions for an attractive invariant set that render a stable regime using a quite simple NN with one hidden layer. Numerical simulations show the performance of the proposed approach. In addition, robustness is analyzed when the robot is subject to Liptchitz disturbances, interestingly showing relaunching of the learning mechanism when needed. Finally, a discussion on dealing with asymptotic stability, robustness issues, and the learning mechanism from a reward provided by the expert user is addressed.