Real-time self-learning for control law adaptation in nonlinear systems using encoded check states

Abstract

With the wide proliferation of autonomous sense-and-control real-time systems (such as robots and self-driven cars), a key research objective is rapid recovery from the effects of anomalies and impairments arising from performance degradation of sensors and actuators and electro-mechanical subsystems due to field wear and tear. This must be achieved with minimal impact on system performance while maintaining low implementation overhead and high coverage of multi-parameter failure mechanisms. In this work, we propose a reinforcement learning framework for on-line control law adaptation in autonomous nonlinear systems assisted by system state encodings. These encodings are exploited to generate time-varying error signals whose (transient) waveforms in relation to the input stimulus, contain root-cause diagnostic information. This establishes a statistical correlation between the transient waveforms and the parameters of the optimal nonlinear controller under arbitrary multi-parameter perturbations of sensor/actuator and subsystem performances. Consequently this correlation is tapped, using pre-deployment supervised learning algorithms, to predict near-optimal controller parameter values whenever sufficiently large parameter deviations are detected (due to non-zero error signals). From these near-optimal starting conditions, an actor-critic reinforcement learning controller for nonlinear systems quickly converges to the optimal control law for the parameter-perturbed system (up to 10× faster than for systems not assisted by the diagnostic information provided by the state encoding driven error signal above). We implement the proposed methodology on two nonlinear systems demonstrating fast performance recovery in real time.