🤖 AI

Robust Shielding for Safe Reinforcement Learning

This paper introduces a novel, sound, and optimal shielding framework for robust Markov decision processes that guarantees the safety of reinforcement learning agents under worst-case transition uncertainties while combining with sampling methods to provide probably approximately correct (PAC) safety guarantees for learned models.

Original authors: Edwin Hamel-De le Court, Thom Badings, Alessandro Abate, Francesco Belardinelli, Francesco Fabiano

Published 2026-06-02

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Edwin Hamel-De le Court, Thom Badings, Alessandro Abate, Francesco Belardinelli, Francesco Fabiano

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to play a video game, like Pac-Man, or to drive a car. You want the robot to learn how to get the highest score or reach its destination as fast as possible. This is called Reinforcement Learning. The robot learns by trying things out: it moves, sees what happens, and gets a "reward" (points) for good moves or a "penalty" for bad ones.

The problem is that to learn, the robot has to explore. It has to try risky moves to see if they work. But in the real world, a risky move could mean a robot crashing into a wall or a self-driving car hitting a pedestrian. We can't let the robot learn by trial and error if the trial could be dangerous.

The Problem: The "Black Box" of Reality

Usually, to keep a robot safe, we need a "shield"—a safety guard that stops the robot from making dangerous moves. But to build a perfect shield, you need to know the exact rules of the world (the physics, the traffic laws, the game mechanics).

In the real world, we don't know the exact rules. We only have some data from past runs or a simulator. If we guess the rules based on limited data, we might be wrong. If our shield is built on a wrong guess, it might fail to stop a disaster.

The Solution: The "Worst-Case" Umbrella

This paper introduces a new kind of shield designed for when we don't know the exact rules. Instead of guessing one set of rules, the authors treat the unknown world as a game between two players:

The Robot (Agent): Trying to get a high score.
The "Gremlin" (Adversary): A mischievous force that tries to make the robot fail by picking the worst possible outcome for every move the robot makes.

The authors call this a Robust MDP (Markov Decision Process). Think of it like this:

Old Way: "Based on my data, there's a 90% chance this bridge holds. I'll let the robot cross." (If the bridge actually breaks, the robot falls).
New Way (This Paper): "I don't know the exact strength of the bridge, but I know it's somewhere between 'weak' and 'strong.' I will build a shield that assumes the bridge is weak (the worst case). If the robot can cross safely even if the bridge is weak, it will definitely be safe if the bridge is strong."

How It Works: The "Safety Budget"

The paper uses a clever trick involving a Safety Budget.

Imagine the robot has a wallet with a specific amount of "Safety Money" (let's say $100). Every time the robot takes a step, there is a tiny risk it might lose some money.

The shield calculates the worst-case chance of losing money for every possible move.
If a move risks losing more money than the robot has left in its wallet, the shield blocks that move.
If the move is safe enough to keep the wallet from going bankrupt, the shield lets the robot take it.

This "wallet" is updated in real-time. As the robot learns more about the world (by gathering more data), the "Gremlin" becomes less scary. The uncertainty shrinks, the "worst-case" scenario becomes less severe, and the robot gets more freedom to take risks that lead to higher rewards.

The "Shield" in Action

The paper describes a three-step process:

Learn the Uncertainty: The robot gathers data from the environment. Instead of saying "The probability of falling is 5%," it says "The probability of falling is between 2% and 8%." This range is the "Robust" part.
Build the Shield: Using these ranges, the shield is built to guarantee safety even if the probability is at the scary end of the range (8%).
Let the Robot Play: The robot plays the game. The shield watches every move. If the robot tries to do something that might be unsafe (even if it's only unsafe in the worst-case scenario), the shield steps in and forces a safer choice.

The Results: Safe but Smart

The authors tested this on games like Pac-Man and a grid-world with "color bombs."

The "Guess" Method (Old Way): If you just guess the rules based on data, the robot often gets a high score but crashes into the ghost or bomb because the guess was slightly wrong.
The "Robust Shield" (New Way):
- At first: When the robot has very little data, the shield is very strict. It says "No, you can't go there, it might be dangerous!" The robot plays very safely but gets a lower score.
- As data grows: As the robot learns more, the "uncertainty range" gets smaller. The shield realizes, "Oh, that move isn't actually that risky!" It relaxes its rules.
- The Outcome: The robot stays 100% safe (it never crashes) but eventually learns to play almost as well as a robot that knew all the rules from the start.

Summary Analogy

Imagine you are teaching a child to ride a bike.

The Old Way: You tell the child, "I think the road is flat, so go ahead and ride fast." If the road actually has a hidden pothole, the child falls.
The New Way (This Paper): You don't know if the road is flat or bumpy. So, you put training wheels on the bike (the Shield). You tell the child, "We will assume the road is full of potholes. If you can ride safely with training wheels on a bumpy road, you are safe."
- At first, the training wheels are heavy, and the child moves slowly.
- But as you drive the bike over the road and realize it's actually smooth, you slowly lift the training wheels.
- The child never falls (Safety Guaranteed), but eventually, they are riding just as fast as if they had known the road was smooth all along.

This paper proves mathematically that this method works: it guarantees the robot won't do anything dangerous, even when we are unsure about the world, and it lets the robot learn to be efficient as it gathers more information.

Technical Summary: Robust Shielding for Safe Reinforcement Learning

Problem Statement

Reinforcement Learning (RL) agents operating in unknown environments, modeled as Markov Decision Processes (MDPs), face a critical safety challenge: standard algorithms explore actions unrestrictedly, inevitably selecting potentially harmful actions. While "shielding" mechanisms exist to formally guarantee safety by restricting agent actions, existing techniques typically require exact knowledge of the transition dynamics relevant to safety. In practical RL scenarios, these dynamics are unknown.

Current approaches to handling unknown dynamics often rely on learning point estimates (e.g., Maximum Likelihood Estimates) of transition probabilities. However, this introduces statistical errors that compromise formal safety guarantees. Alternatively, learning uncertainty sets around transition probabilities (e.g., using Probably Approximately Correct (PAC) bounds) is a viable strategy, but existing shielding frameworks are incompatible with such sets, as they are designed for models with exact probabilities. This paper addresses the gap in providing a shielding framework that guarantees safety for RL agents operating in unknown MDPs by leveraging Robust MDPs (RMDPs).

Methodology

The authors propose a novel shielding framework for RMDPs, where transition probabilities are represented as sets (uncertainty sets) rather than single values. The methodology proceeds in two main stages:

1. Formal Framework for Robust MDPs

The paper defines an RMDP as a game between an agent (choosing actions) and an adversary (choosing transition probabilities from uncertainty sets). Safety is defined via Linear Temporal Logic (LTL) formulas, specifically focusing on safety fragments where a "bad prefix" leads to a violation.

Shield Definition: A shield is defined as a pair $(D, \Gamma)$ , where $D$ is a controllable monitor (extending standard monitors with auxiliary actions) and $\Gamma$ maps observation states and monitor states to a set of allowed probability distributions over actions.
Construction: The shield is constructed on a finite safety abstraction of the RMDP. It utilizes an inductive value function $\beta$ that serves as a robust upper bound on the probability of reaching a "bad" state (violating the LTL formula). The shield restricts the agent to actions where the worst-case expected next "violation budget" remains within the current budget.
Theoretical Guarantees: The authors prove that for a given RMDP, the constructed shield is:
- Sound: Every policy admissible by the shield satisfies the safety specification under the worst-case adversary.
- Optimal (Approximate): Every safe policy of the RMDP is admissible by the shield (up to an $\epsilon$ error depending on the approximation of the value function). If the value function is exact, the shield is exactly optimal.

2. Shielding Unknown MDPs via Learning

The framework is instantiated for unknown MDPs through a learning pipeline:

Learning RMDPs: Transition probabilities are learned from sampled transitions using Clopper-Pearson confidence intervals to construct PAC uncertainty sets. This results in a learned RMDP $\hat{M}$ that "contains" the true MDP $M$ with high confidence ( $1 - \delta$ ).
Shield Synthesis: A shield is computed on the learned RMDP $\hat{M}$ using the robust value iteration method described above.
Deployment: The shield is deployed on the true (unknown) MDP. The agent interacts with the environment, but its action choices are constrained by the shield derived from $\hat{M}$ .

The paper establishes that if the learned RMDP contains the true MDP with high confidence, the shield constructed on the RMDP guarantees safety for the true MDP with the same confidence. Furthermore, as the number of samples increases, the uncertainty sets tighten, and the shield's conservativeness decreases, allowing the agent to approach the optimal reward of the true MDP.

Key Contributions

Novel Shielding Framework for RMDPs: The paper introduces the first shielding framework specifically designed for RMDPs (MDPs with sets of transition probabilities), addressing the limitation of existing methods that require exact dynamics.
Soundness and Optimality Proofs: The authors provide rigorous proofs demonstrating that their shield is sound (guarantees safety under worst-case dynamics) and approximately optimal (allows for near-optimal rewards) for RMDPs.
Integration with PAC Learning: The work combines robust shielding with existing sampling methods to learn transition probabilities with PAC guarantees. This enables the construction of shields for unknown MDPs that provide high-confidence safety guarantees.
Empirical Validation: The approach is evaluated on multiple environments (Media Streaming, Color Bomb Gridworld, Pacman) comparing the proposed method against baselines using exact MDPs and Maximum Likelihood Estimation (MLE).

Results

Experimental results demonstrate the trade-off between data availability and shield conservativeness:

Safety Guarantees: The proposed robust shields (Kno and Unk settings) consistently satisfy the required safety specification probability ( $1-p$ ) across various sample sizes, whereas the MLE-based shield frequently violates safety thresholds, particularly with small to moderate sample sizes.
Performance Recovery: As the number of samples increases, the uncertainty sets in the learned RMDP become tighter. Consequently, the robust shields become less restrictive, and the expected return of the protected agent converges toward the performance of an agent operating with full knowledge of the MDP (the MDP baseline).
Handling Unknown Graphs: The framework successfully handles cases where the MDP graph structure is unknown (Unk setting), provided a minimum transition probability is assumed, though this requires a higher minimum sample size to learn the graph structure reliably.

Significance and Claims

The paper claims to provide a formal solution to the problem of safe exploration in RL when transition dynamics are unknown. By shifting from point estimates to uncertainty sets (RMDPs), the authors argue that it is possible to construct shields that offer formal, high-confidence safety guarantees without requiring perfect knowledge of the environment.

The significance lies in bridging the gap between formal methods (which typically assume known models) and data-driven RL (which operates on unknown models). The authors emphasize that while robust methods may be conservative with limited data, this conservatism is a necessary trade-off to ensure safety. As data accumulates, the method naturally relaxes, enabling the agent to achieve strong expected returns while maintaining safety. This contrasts with state-of-the-art approaches using point estimates, which the paper demonstrates can lead to unsafe behavior due to unmodeled statistical uncertainty.

The work positions itself as a step toward "safe autonomy" in safety-critical scenarios where model uncertainty is inherent, offering a theoretically grounded alternative to heuristic safety constraints or methods lacking finite-sample guarantees.