⚡ electrical engineering

Robust Koopman Control Barrier Filters for Safe Actor-Critic Reinforcement Learning

This paper introduces Robust Koopman-CBF SAC, a safe reinforcement learning framework that learns a data-driven Koopman predictor to construct and enforce tightened control barrier function constraints via a quadratic program, thereby achieving zero constraint violations in benchmarks while balancing task performance and safety.

Original authors: Dhruv S. Kushwaha, Zoleikha A. Biron

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dhruv S. Kushwaha, Zoleikha A. Biron

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to walk or balance a pole on its head. You want the robot to get really good at the task (high performance), but you also need to make sure it never falls over, hits a wall, or breaks its own joints (safety).

This paper presents a new method called Robust Koopman-CBF SAC that acts like a "smart safety guard" for robots learning through trial and error. Here is how it works, broken down into simple concepts:

1. The Problem: The "Wild Explorer" vs. The "Strict Guardian"

The Explorer (The Robot): To learn, a robot needs to try new things. It might try a crazy move that works great, or it might try a move that causes it to crash. Standard learning algorithms are like wild explorers; they are great at finding the best way to do a task, but they don't naturally know how to avoid crashing.
The Guardian (The Safety Filter): Traditional safety systems are like strict guardians. They know the rules (e.g., "don't go past this line") and stop the robot if it tries to break them. However, these guardians usually need a perfect manual of how the robot moves (a physics textbook) to do their job. If the robot is complex or the physics are unknown, the guardian gets confused and stops working.

2. The Solution: A "Translator" and a "Safety Net"

The authors created a system that combines the best of both worlds using three main tricks:

A. The Translator (Koopman Lifting)

Robots often move in complicated, non-linear ways (like a swinging pendulum). It's hard to predict exactly where they will go next.

The Analogy: Imagine trying to predict the path of a swirling leaf in the wind. It's chaotic. But if you translate that chaotic motion into a different "language" (a higher-dimensional space), the leaf's path suddenly looks like a straight line.
How it works: The system uses a mathematical "translator" (Koopman operator) to convert the robot's messy, real-world movements into a simplified, linear language. In this new language, predicting the future is easy, like drawing a straight line on a piece of paper.

B. The Safety Net with a "Buffer Zone" (Robust CBF)

Even with a translator, the prediction isn't perfect. There is always a little bit of "static" or error.

The Analogy: Imagine a tightrope walker. If you tell them, "Stay exactly on the wire," they might fall if there's a tiny gust of wind. But if you tell them, "Stay within this wide, safe zone around the wire," they are much safer.
How it works: The system doesn't just guess the robot's future; it measures how wrong its guesses have been in the past (the "residual"). It then adds a buffer zone (a margin of error) to the safety rules. If the robot tries to move, the system checks: "Even if my prediction is slightly wrong, will the robot still be safe?" If the answer is yes, it lets the move happen. If the answer is no, it gently nudges the robot back to safety.

C. The Coach (Actor-Critic Learning)

The robot learns using a "Coach" system (Soft Actor-Critic).

The Twist: Usually, the coach tells the robot what to do, and the robot does it. But here, the "Safety Guardian" might change the robot's action before it happens.
The Innovation: The authors made sure the Coach learns from what the robot actually did (after the safety guard fixed it), not just what it wanted to do. This prevents the Coach from getting confused and teaching the robot bad habits.

3. What They Found (The Results)

The team tested this on two types of robots: simple ones and complex, real-world-like ones.

The Simple Robots (CartPole & Quadrotor):
- The Result: The system was perfect. It achieved zero crashes while still performing the task just as well as an unsafe robot.
- Why: The "translator" worked great for these simple movements, and the "buffer zone" was just the right size. The safety guard rarely had to step in because the robot learned to stay safe on its own.
The Complex Robots (Walking Robots like HalfCheetah and Walker):
- The Result: The system helped reduce crashes, but it wasn't perfect. It couldn't stop the robots from falling as effectively as other methods in some cases.
- Why: These robots have "feet" that hit the ground, creating sudden, jerky movements (like a car hitting a pothole). The "translator" couldn't simplify these jerky movements well enough. The "buffer zone" became too big to be useful, or the safety rules became impossible to follow.
- The Insight: The authors discovered a "warning light." Before even starting the training, they could measure the "prediction error" (the static in the translator). If this error was too high, they knew the safety system wouldn't work well for that specific robot. This is a major finding: You can predict if a safety filter will work just by checking the math beforehand.

Summary

This paper introduces a smart safety layer for learning robots. It translates complex movements into simple ones, adds a safety buffer to account for math errors, and teaches the robot based on its actual safe actions.

When it works: It is incredibly effective for robots with smooth, predictable movements (like balancing a pole), offering perfect safety without slowing down learning.
When it struggles: It hits a wall with robots that have sudden, jerky impacts (like walking or running), because the math struggles to predict those sudden changes.
The Takeaway: The authors provide a tool to check before you start if your safety system will work. If the robot's movements are too chaotic for the math to simplify, this specific safety filter might not be the right tool for the job.

Technical Summary: Robust Koopman Control Barrier Filters for Safe Actor-Critic Reinforcement Learning

Problem Statement
Safe reinforcement learning (RL) for robotic systems faces a fundamental tension: improving task performance while strictly satisfying state and input constraints during both training and deployment. While Control Barrier Functions (CBFs) offer a principled mechanism for enforcing forward invariance via minimally invasive safety filters, their application in model-free RL is hindered by the requirement for accurate system dynamics and hand-designed barrier certificates. Existing constrained RL methods often provide only expected cumulative cost guarantees rather than pointwise safety, and reward penalties lack step-wise guarantees. The authors address the challenge of combining the sample efficiency of off-policy deep RL with a data-driven CBF safety filter that remains convex for nonlinear systems, specifically accounting for the approximation errors inherent in finite-dimensional Koopman models.

Methodology: Robust Koopman-CBF SAC
The paper proposes Robust Koopman-CBF SAC, a safety-filtered actor-critic framework that integrates three core components:

Data-Driven Koopman Lifting: The method learns a finite-dimensional Koopman predictor from offline rollout data using Extended Dynamic Mode Decomposition (EDMD). Nonlinear dynamics are lifted into a space of observables $\psi(x)$ where the evolution is approximately linear: $z_{t+1} \approx Az_t + Bu_t$ . The dictionary $\psi$ includes original state coordinates and Radial Basis Functions (RBFs) to ensure physical constraints can be represented as affine functions in the lifted space.
Robust Lifted CBF Constraints: To handle the approximation error of the finite-dimensional model, the authors introduce a projected residual margin ( $\rho$ ). Instead of bounding the full residual norm, they estimate the component of the prediction error that directly affects the barrier inequality ( $|c^\top r|$ ) using a held-out calibration set. This margin is computed via empirical or conformal quantiles. The safety condition is tightened to:
$h_K(Az_t + Bu_t) \geq (1 - \eta)h_K(z_t) + \rho$
This results in an affine constraint on the control input, allowing the safety filter to be implemented as a lightweight Quadratic Program (QP).
Actor-Critic Integration: The framework modifies the standard Soft Actor-Critic (SAC) algorithm to resolve the mismatch between nominal and executed actions:
- Critic Training: The critic is trained on the executed safe action ( $u^{safe}_t$ ) generated by the QP filter, ensuring value estimates reflect the actual closed-loop dynamics.
- Actor Regularization: The actor receives a differentiable CBF proposal penalty ( $\ell_{cbf}$ ) that encourages it to propose actions already within the feasible set, reducing reliance on the filter over time.
- Slack Handling: The QP includes slack variables to maintain numerical feasibility under actuator limits, though the authors note that positive slack weakens formal safety certificates.

Key Contributions

Robust Filter Formulation: Introduction of a robust Koopman-CBF safety filter for model-free actor-critic RL that lifts nonlinear dynamics to an approximately linear system and enforces safety via an affine QP constraint.
Projected Residual Margin: A novel method for estimating finite-dimensional Koopman model error using a projected residual margin derived from calibration data, yielding a directly implementable robust discrete-time CBF condition.
Consistent Actor-Critic Training: Integration of the filter with SAC using executed-action critic learning and an actor-side feasibility penalty, eliminating the common mismatch where the policy optimizes for unfiltered actions while the environment transitions are generated by filtered actions.
Empirical Diagnostics: A comprehensive evaluation demonstrating that the projected residual margin $\rho$ serves as a pre-training computable diagnostic for filter effectiveness.

Experimental Results
The method was evaluated on Safe-Control-Gym (CartPole, Quadrotor 2D) and Safety Gymnasium (Walker, HalfCheetah).

Low-Dimensional Tasks (CartPole & Quadrotor): On CartPole stabilization and tracking, Robust Koopman-CBF SAC achieved zero constraint violations across all seeds while matching or exceeding the returns of unconstrained SAC. On Quadrotor 2D tracking, it reduced violations by 96.8% (from 12.5% to 0.4%). These successes were attributed to small projected residual margins ( $\rho \approx 10^{-3}$ ) and correct relative degree handling (using composite barriers for the Quadrotor).
High-Dimensional Locomotion (Walker & HalfCheetah): The results were mixed, exposing structural limitations. On Walker, violations were reduced from 77.8% to 37.0%, but the method was outperformed by SAC+Lagrangian (3.34% violations). On HalfCheetah, violations remained high (82.8%) with a severe performance penalty (-70% return).
Failure Analysis: In high-dimensional, contact-rich tasks, the projected residual margin $\rho$ was large (0.698 for Walker, 1.782 for HalfCheetah). This large error caused the robust CBF constraint to either become trivially satisfied (when the agent was already unsafe) or infeasible under actuator limits, rendering the filter ineffective. The authors identify that linear EDMD models cannot capture the non-smooth velocity transitions caused by ground contact.

Significance and Claims
The paper claims that Robust Koopman-CBF filters provide a promising bridge between model-free RL and certifiable safety, particularly for tasks where the lifted model and barrier align well with the system structure. The primary significance lies in the diagnostic capability of the projected residual margin $\rho$ : it is a pre-training metric that predicts whether a Koopman-CBF filter will be effective.

The authors are modest regarding the method's universality, explicitly stating that the approach has structural limitations in contact-rich, high-dimensional environments where first-order linear approximations fail. They clarify that while the method offers pointwise safety when $\rho$ is small and the QP is feasible, it does not guarantee safety when the residual error dominates the barrier margin or when the relative degree of the barrier is mismatched. The work highlights the need for high-order or multi-step Koopman extensions to address these limitations in complex locomotion tasks.