A Review On Safe Reinforcement Learning Using Lyapunov and Barrier Functions

Problem Statement

Reinforcement Learning (RL) has demonstrated efficacy in solving complex decision-making problems across robotics, energy management, and autonomous systems. However, standard RL approaches lack rigorous theoretical guarantees regarding closed-loop stability and constraint satisfaction. In safety-critical applications, constraint violations can lead to system failure or physical harm. While control-theoretic methods offer certificates for stability (via Lyapunov functions) and safety (via Barrier functions), integrating these guarantees into data-driven RL frameworks remains a significant challenge. The core problem addressed is how to ensure that a computed RL policy maintains system stability and adheres to operational constraints during both training and deployment, particularly in the absence of perfect system models.

Methodology and Scope

This paper provides a comprehensive review of Safe Reinforcement Learning (Safe RL) techniques that explicitly utilize Control Lyapunov Functions (CLFs) and Control Barrier Functions (CBFs). The authors categorize the literature based on the role these certificates play within the RL loop:

1. Lyapunov-only approaches: Using CLFs to guarantee asymptotic stability.
2. Barrier-only approaches: Using CBFs to guarantee forward invariance of safe sets.
3. Combined approaches: Integrating both CLFs and CBFs to simultaneously ensure stability and safety.

The review distinguishes between model-based formulations (relying on known or approximated dynamics) and model-free formulations (relying purely on data), noting a decisive shift in the field toward model-free and combined approaches post-2017, with a surge in combined CLF–CBF methods after 2022.

The review analyzes how certificates are embedded into RL algorithms, including:

• Reward/Cost Shaping: Modifying the objective function to penalize instability or constraint violations.
• Critic/Value Function Parameterization: Treating the value function itself as a Lyapunov candidate.
• Policy Constraints: Using Quadratic Programming (QP) filters to project nominal RL actions onto safe sets.
• Loss Regularization: Adding penalty terms to actor-critic losses to enforce certificate conditions.
• Formal Verification: Using counterexample-guided synthesis (CEGIS) or Sum-of-Squares (SOS) programming to verify neural certificates.

Key Contributions

The paper makes three primary contributions to the existing literature:

1. Taxonomy and Classification: It offers a structured classification of Safe RL methods, moving beyond broad surveys to focus specifically on the integration of Lyapunov and Barrier functions. It details how these functions are utilized as rewards, critics, constraints, or safety filters.
2. Identification of Open Problems: The authors define specific, per-class open problems that hinder broader deployment:
   • For Lyapunov methods: Certificate validity under function approximation and distribution shift.
   • For Barrier methods: Feasibility and deadlock issues under hard CBF–QP shielding, particularly when constraints conflict.
   • For Combined methods: Joint CLF–CBF feasibility under model uncertainty.
3. Scalability Analysis: The review highlights that deployment to high-dimensional and partially observable settings remains the dominant scalability barrier across all three classes. It contrasts the computational complexity of different approaches, noting that while QP-based filters offer hard safety guarantees, they suffer from the "curse of dimensionality" and feasibility issues in high-dimensional spaces.

Results and Comparative Analysis

The review synthesizes findings from numerous studies (summarized in Tables 3, 5, and 7) to draw several comparative conclusions:

• Safety Enforcement Strength: Combined CLF–CBF–QP filters provide the strongest practical safety (hard invariance + asymptotic stability) when feasible. Pure CBF-QP filters offer hard invariance only. CMDP-Lagrangian methods and reward shaping provide "soft" safety (expectation-level or penalty-based), which is less robust but more scalable.
• Model Dependence: CBF and combined methods typically require control-affine structures and Lie derivative computations, often necessitating model learners (e.g., Gaussian Processes) or observers. Lyapunov-as-value methods can be more model-free but rely heavily on the validity of the learned critic.
• Sample Efficiency: Hard execution-time filters (CBF-QP) generally require fewer samples to reach safety thresholds because they prevent unsafe rollouts. Reward shaping accelerates convergence but provides weaker sample-efficiency gains.
• Benchmarking: The authors identify Safe-Control-Gym as the most suitable benchmark for these methods because it provides symbolic models for analytic Lie derivatives, constraint APIs, and disturbance injection, which are often missing in standard environments like MuJoCo.

Significance and Claims

The paper positions itself as an RL-centric synthesis rather than a general survey of safe learning for control. Its significance lies in:

• Bridging Theory and Practice: It demonstrates the promising scope for providing safety guarantees for complex dynamical systems using both model-based and model-free RL.
• Highlighting Recent Shifts: It documents the field's transition from model-based to model-free formulations and the emergence of combined CLF–CBF approaches as the most active sub-area post-2022.
• Defining Future Directions: The authors argue that while theoretical guarantees exist, the gap between theory and deployment is centered on scalable certificate discovery, robust generalization under distribution shift, and handling partial observability.

The review concludes that while Lyapunov and barrier functions provide a principled route to adding stability and constraint-satisfaction structure to learning-based control, the remaining challenges involve resolving feasibility barriers in constrained optimization filters and establishing robust learning principles for data-driven certificates in high-dimensional, real-world settings.