Hybrid safe reinforcement learning: Tackling distribution shift and outliers with the Student-t’s process

摘要

Safe reinforcement learning (SRL) aims to optimize control policies that maximise long-term reward, while adhering to safety constraints. SRL has many real-world applications such as, autonomous vehicles, industrial robotics, and healthcare. Recent advances in offline reinforcement learning (RL)—where agents learn policies from static datasets without interacting with the environment—have made it a promising approach to derive safe control policies. However, offline RL faces significant challenges, such as covariate shift and outliers in the data, which can lead to suboptimal policies. Similarly, online SRL, which derives safe policies through real-time environment interaction, struggles with outliers and often relies on unrealistic regularity assumptions, limiting its practicality. This paper addresses these challenges by proposing a hybrid-offline-online approach. First, prior knowledge from offline learning guides online exploration. Then, during online learning, we replace the popular Gaussian Process (GP) with the Student-t’s Process (TP) to enhance robustness to covariate shift and outliers. • Tail decay rates of Gaussian and Student-t processes’ densities are explicitly derived. • A novel TP-based robust hybrid offline-online SRL algorithm maximizes adaptability. • Using t-posterior variance to infer safety under covariate shift and outliers is novel in SRL. • A novel criterion evaluates two SRL agents, identifying the’safer’ w.r.t safety constraints.