Adaptive Linear Quadratic Control of Unknown Linear Time-Varying Systems via Policy Gradient Methods

Abstract

Unknown linear time-varying (LTV) systems require the control policy to adapt from online closed-loop data as dynamics evolve. Existing methods usually update the policy by solving a one-shot optimization problem, which can be computationally demanding and sensitive to noisy model estimates. In this paper, we propose a policy gradient adaptive control (PGAC) method for LTV system control with unknown model parameters. Specifically, PGAC integrates online policy optimization into feedback by updating the state-feedback policy with one-step gradient descent of the linear quadratic regulator cost at each time instant. This incremental update is computationally light and naturally limits policy variations caused by noisy data. To explicitly compute the policy gradient online, we estimate local models from recent closed-loop trajectories using normalized sliding-window least-squares. We provide stability and convergence certificates of PGAC for two classes of LTV systems. For slowly time-varying systems, we prove that the closed-loop system achieves practical exponential stability without a dwell-time condition. For piecewise-constant LTV systems, we establish practical stability through a dwell-time contraction argument. We also provide average frozen-time optimality-gap bounds of the policy sequence for both classes. Finally, we validate the effectiveness of our method via numerical case studies of both LTV and nonlinear systems.