⚡ electrical engineering

Provably Safe, Yet Scalable Reinforcement Learning

This paper introduces PS2-RL, a novel two-phase framework that achieves provably safe and scalable reinforcement learning by training a learned backup policy to generate implicit control-invariant sets online, which are then enforced via a differentiable projection layer without restricting the underlying RL algorithm.

Original authors: Kai S. Yun, Zeyang Li, Navid Azizan

Published 2026-06-15

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Kai S. Yun, Zeyang Li, Navid Azizan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Wild Child" vs. The "Strict Guardian"

Imagine you are teaching a robot (or a self-driving car) to do a difficult task, like driving a race car around a track or flying a drone through a storm. You want the robot to be fast and skilled (high performance), but you also need it to be 100% safe (never crash or hit a wall).

The "Wild Child" (Standard AI): Current AI methods are like wild children. They learn by trial and error. They get really good at the task, but they might accidentally crash because they haven't been formally taught the rules of physics. They are "empirically safe" (they haven't crashed yet), but there is no guarantee they won't crash tomorrow.
The "Strict Guardian" (Old Safety Methods): Other methods try to force safety by building a rigid cage around the robot. They calculate every possible safe path in advance. The problem is, for complex robots (like a 10-dimensional drone), calculating this cage takes forever. To make it work, they make the cage so small and conservative that the robot can barely move. It's safe, but it's useless because it's too slow and stiff.

The Goal: We need a robot that is as skilled as the "Wild Child" but as safe as the "Strict Guardian," without the slow, rigid cage.

The Solution: PS2-RL (The "Two-Phase" Team)

The authors propose a new framework called PS2-RL. Think of it as a two-person team working together: a Recovery Expert and a Performance Athlete.

Phase 1: Training the "Recovery Expert" (The Backup Plan)

Before the robot tries to do the main task, it first learns a "backup plan."

The Analogy: Imagine a gymnast learning how to fall safely. If they slip, they know exactly how to twist their body to land on their feet and stop rolling.
How it works: The AI learns a "Safe-Arrival Policy." This isn't about winning the race; it's about knowing how to steer the robot from anywhere in the safe zone back to a tiny, super-safe "home base" (like a parking spot) without hitting anything.
The Innovation: Old methods used simple, pre-written math formulas for this (like a basic LQR controller). The paper uses AI to learn a smarter recovery plan. This allows the robot to recover from much more dangerous situations than before, effectively making the "safe zone" much bigger.

Phase 2: Training the "Performance Athlete" (The Main Task)

Now that the robot has a super-smart backup plan, we train it to do the actual job (like flying the loop-the-loop).

The Analogy: Imagine a Formula 1 driver. They are allowed to drive as fast and aggressively as they want, but they have a "Safety Filter" attached to their steering wheel.
How it works: The AI tries to drive fast. If it tries to make a move that would cause a crash, the Control-Invariant Layer (the Safety Filter) instantly intercepts the command. It doesn't stop the car; it just nudges the steering wheel slightly to the nearest safe angle that keeps the car on the track.
The Magic: Because this filter is "differentiable" (mathematically smooth), the AI can learn through it. The AI learns that "If I turn too hard here, the filter will correct me, and I'll lose time." So, the AI learns to drive right up to the edge of safety without ever crossing it, maximizing speed while staying safe.

Why This is a Big Deal

1. It Scales (It works for big, complex robots)
Old safety methods tried to map out the entire "safe universe" for the robot before it started moving. For a robot with 10 moving parts (like a drone with position, speed, and rotation), this is like trying to draw a map of every possible path in a city the size of the moon. It's impossible.

PS2-RL's trick: Instead of mapping the whole city, it just asks, "If I go this way, can my Backup Expert get me home?" It calculates this on the fly. This makes it fast enough to work on complex, high-dimensional robots.

2. It's "Provably Safe" (No guessing)
Many AI safety methods say, "We think it's safe." PS2-RL says, "We know it's safe."

The Guarantee: Because the system is built on a mathematical foundation (Backup Control Barrier Functions), the authors can prove mathematically that as long as the robot starts in a safe spot, it will never leave the safe zone, no matter how crazy the task is.

3. It's Not Conservative (It doesn't hold back)
Because the "Recovery Expert" learned a smart way to get home, the "Safety Filter" doesn't have to be so strict. The robot can take risks and drive aggressively because it knows it has a better safety net than before.

The Results: The Experiments

The authors tested this on two scenarios:

A Unicycle (Lane Keeping): A simple robot trying to stay in a lane while following a wiggly path that goes outside the lane.
- Result: PS2-RL stayed in the lane 100% of the time and tracked the path much better than other methods.
A Quadrotor (Drone Powerloop): A 10-dimensional drone trying to fly a vertical loop while doing a flip, where the top of the loop is dangerously close to a "ceiling" it must not hit.
- Result: This is a very hard task. Other methods either crashed or flew very slowly to be safe. PS2-RL flew the loop perfectly, stayed under the ceiling 100% of the time, and was significantly faster and more accurate than the next best method.

Summary

PS2-RL is like giving a race car driver a smart co-pilot. The driver (the AI) pushes the car to the limit to win the race. The co-pilot (the learned backup policy) watches the road and instantly steers the car back to safety if the driver gets too close to the edge. The result is a car that is both fast and guaranteed not to crash, even on a track that is too dangerous for anyone else to drive.

Technical Summary: Provably Safe, Yet Scalable Reinforcement Learning (PS2-RL)

Problem Statement

Safe Reinforcement Learning (RL) aims to learn policies that maximize task rewards while strictly satisfying safety constraints. Existing approaches generally fall into two categories, both with significant limitations:

Soft-Constrained Optimization: Methods using Lagrangian relaxation or neural certificate functions often achieve empirical success but lack formal guarantees. Consequently, they cannot prevent catastrophic failures during deployment.
Strictly Guaranteed Methods: Approaches relying on explicit certificate functions (e.g., Control Barrier Functions or Safety Indices) provide formal safety but require the synthesis and verification of control-invariant sets. This process scales poorly with state dimension, often becoming computationally intractable for high-dimensional systems and yielding overly conservative behaviors.

The core challenge is to develop an RL framework that provides provable safety guarantees for high-dimensional, input-constrained systems without sacrificing performance or scalability.

Methodology: The PS2-RL Framework

The authors propose PS2-RL (Provably Safe, yet Scalable RL), a two-phase architecture that avoids explicit synthesis of global invariant sets. Instead, it leverages a learned backup policy to generate an implicit control-invariant set online.

Phase I: Learning the Safe-Arrival Policy

The first phase focuses on expanding the region of safety. The system is anchored by a small, certified base set $B$ (e.g., an LQR-stabilized equilibrium) and a base controller $\pi_B$ . The goal is to learn a safe-arrival policy ( $\pi_{SA}$ ) that steers the system from a larger state space into $B$ while remaining within the safe set $S$ .

Safe-Arrival Value Function: The authors introduce a novel objective function, the safe-arrival value function, which acts as an indicator-based reward. It quantifies the ability of a policy to reach $B$ $B$ safely before entering a failure set $F$ $F$ .
- It satisfies a self-consistency condition and a Bellman equation, allowing standard RL pipelines (e.g., TD3, SAC) to train the policy.
- The objective encourages time-optimal safe recovery, effectively enlarging the Backup-Induced Control-Invariant Set ( $C_T(\pi_b)$ ). This set consists of all states from which the composed backup policy ( $\pi_b = \pi_{SA}$ outside $B$ , $\pi_B$ inside $B$ ) can reach $B$ within a horizon $T$ without violating safety.

Phase II: End-to-End Training with a Control-Invariant Layer

The second phase trains the task-specific RL policy ( $\pi_\phi$ ) to maximize rewards while strictly adhering to the safety constraints induced by the fixed backup policy from Phase I.

Control-Invariant Layer (CIL): The authors construct a differentiable projection layer that maps the nominal output of the RL policy to the BCBF-admissible input set ( $K_{BCBF}$ $K_{B C B F}$ ).
- This set is defined by the Backup Control Barrier Function (BCBF) constraints, which are affine in the control input $u$ given a fixed backup flow.
- The layer solves a Quadratic Program (QP) to find the control input closest to the nominal policy's output that satisfies the safety constraints.
Differentiability: Crucially, the QP solver is differentiable. This allows gradients to propagate end-to-end through the safety projection during training. The RL policy learns to optimize performance within the certified safe set, rather than learning a policy that is later patched by a non-differentiable shield.

Key Contributions

PS2-RL Framework: A two-phase architecture that achieves formal safety guarantees for high-dimensional systems without synthesizing explicit global invariant sets. It relies on learned backup rollouts and a control-space projection.
Safe-Arrival Value Function: A novel objective that encodes time-optimal safe recovery to a target set. It admits a Bellman equation, enabling the training of neural backup policies that empirically yield significantly larger safe regions than analytic alternatives (e.g., LQR).
Control-Invariant Layer: A differentiable projection layer that enforces BCBF constraints. The authors prove that this layer preserves the universal approximation property of the underlying neural network, ensuring that safety enforcement does not compromise the policy's expressiveness.
Theoretical Guarantees: Comprehensive proofs establishing that:
- The composed PS2 policy belongs to the class of safe policies ( $\Pi_{safe}$ ).
- The framework solves a certified inner approximation of the original safe RL problem.
- The projection layer maintains the approximation power of the policy network.

Experimental Results

The framework was evaluated on two tracking tasks with intentionally unsafe reference trajectories:

Unicycle Lane-Keeping (3D state): PS2-RL achieved 100% safety across all seeds and outperformed all baselines in tracking accuracy (lateral and heading RMSE) while maintaining safety. It significantly reduced conservatism compared to using an analytic backup policy (PS2ABP).
Quadrotor Powerloop (10D state): This task involved a 10-dimensional system with tight actuation limits and a high-relative-degree altitude constraint.
- PS2-RL achieved 100% safety, whereas methods relying on expected cost constraints (CMDP) or soft penalties frequently violated safety.
- Compared to baselines with formal safety mechanisms (CBF-RL, Model Predictive Shielding), PS2-RL demonstrated substantially lower tracking errors (approx. 40–70% reduction in RMSE).
- The learned safe-arrival policy increased the recoverability of task-relevant near-ceiling states from 60.9% (analytic) to 85.9%.

Ablation Studies: Removing the Control-Invariant Layer at deployment caused safety to drop to 0%, confirming the layer is essential for guarantees. Conversely, using the layer only as a post-hoc filter (without end-to-end training) resulted in 43–53% worse tracking performance, highlighting the necessity of training through the differentiable projection.

Significance and Claims

The paper claims that PS2-RL successfully bridges the gap between provable safety and scalability. By replacing the intractable synthesis of global invariant sets with learned backup rollouts and a differentiable control-space projection, the framework enables safe RL for high-dimensional systems (up to 10 states in the experiments) where prior provably safe methods struggle or become impractical.

The authors emphasize that their approach imposes no restrictions on the underlying RL algorithm and can be integrated into existing training pipelines. Theoretical results confirm that the method provides hard safety guarantees while retaining the high performance and expressiveness required for complex robotic control tasks.

Limitations Acknowledged: The formal guarantees rely on the availability of a control-affine analytic dynamics model or a differentiable simulator. The authors note that extending the framework to uncertain dynamics and perception-based constraints, as well as hardware deployment, remains future work.