🤖 machine learning

Adaptive Probabilistic Shielding by Learning MDPs for Safe Reinforcement Learning

This paper proposes an adaptive probabilistic shielding framework for safe reinforcement learning that integrates online model learning to dynamically compute and update safety constraints as transition probabilities are estimated from an initially unknown MDP.

Original authors: Astrid Horn Brorholt (Aalborg University, Aalborg, Denmark), Maris F. L. Galesloot (Radboud University, Nijmegen, Netherlands), Nils Jansen (Radboud University, Nijmegen, Netherlands), Kim Guldstrand

Published 2026-08-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Astrid Horn Brorholt (Aalborg University, Aalborg, Denmark), Maris F. L. Galesloot (Radboud University, Nijmegen, Netherlands), Nils Jansen (Radboud University, Nijmegen, Netherlands), Kim Guldstrand Larsen (Aalborg University, Aalborg, Denmark), Christian Schilling (Aalborg University, Aalborg, Denmark)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, machines learn to make decisions by trying things out, much like a child learning to walk by falling and getting back up. This process, known as reinforcement learning, allows software agents to discover the best way to solve problems by interacting with an environment and receiving rewards for good choices. However, in the real world, the cost of a wrong step can be catastrophic. A self-driving car cannot afford to test a dangerous maneuver to see if it works; a medical robot cannot risk harming a patient while it figures out the safest path. This creates a fundamental tension: to learn effectively, an agent must explore, but to remain safe, it must avoid the unknown. For years, researchers have tried to solve this by building a "shield"—a safety guard that blocks any action the agent might take that could lead to disaster. The problem is that traditional shields require a perfect, pre-existing map of the world, including exact probabilities for every possible outcome. In the messy reality of the physical world, such a perfect map rarely exists before the agent starts learning.

A team of researchers has developed a new way to handle this dilemma, creating a system where the safety shield learns and adapts alongside the agent. Instead of waiting for a perfect map, their approach assumes the agent knows the general layout of the environment—the possible places it can go and the actions it can take—but does not know the exact odds of where those actions will lead. As the agent explores, it gathers data on what actually happens when it moves. A computer program then uses this fresh data to build an estimate of the world's rules, filling in the missing probabilities. From this estimate, the system constructs a safety shield. Initially, because the data is scarce, the shield is very cautious, blocking many actions to be sure. But as the agent gathers more experience, the estimate becomes sharper, the uncertainty shrinks, and the shield relaxes its grip, allowing the agent to take more efficient risks. This creates a partnership where the agent and its safety guard improve together, ensuring the agent stays safe while it learns how to be effective.

The researchers tested this idea in several simulated worlds, ranging from an aircraft avoiding a collision with another plane to an ant navigating around a predator. In these environments, the agent had to reach a goal while avoiding specific hazards, all while the exact probabilities of slipping or failing were hidden from it. They compared their adaptive method against two other approaches: one where the agent had no safety guard at all, and another where the agent had a perfect shield based on a known, complete map of the world. The results showed that the adaptive method was remarkably successful. It learned policies that were nearly as safe as the perfect shield, while avoiding the frequent, dangerous crashes that occurred when the agent had no guard at all. In some cases, the adaptive system even found better routes than the perfect shield, because its cautious early exploration led it to discover profitable paths that the static, pre-calculated shield had missed.

A key finding was that the system works best when the safety shield is updated regularly as new data comes in. If the shield is updated too rarely, the agent remains stuck in a state of over-caution, unable to take advantage of the knowledge it has gained. If it is updated too often, the system spends too much time recalculating safety rather than learning. The researchers found a balance where updating the shield every thousand learning episodes allowed the agent to refine its understanding of the world without stalling its progress. They also discovered that the way the system estimates probabilities matters. Some methods that assume the worst-case scenario for unknown data tend to be overly restrictive, while methods that are slightly more optimistic allow the agent to explore more freely. The most effective approach used a balanced strategy that remained safe but did not block potentially useful actions simply because the data was not yet perfect.

The study also highlighted a subtle but important detail about how the agent explores. When the agent decides to try a random action to learn something new, the system allows it to choose from all possible actions, even those the current shield considers unsafe. This might seem counterintuitive, but it prevents the agent from getting trapped in a small, safe corner of the world. By occasionally venturing beyond the shield's current limits, the agent gathers the very data needed to prove that those limits can be safely expanded. Without this freedom to explore the edges of the unknown, the shield would never learn to become less conservative. The researchers observed that in complex environments, this willingness to take calculated risks was essential for finding the best possible path.

Ultimately, this work demonstrates that safety and learning do not have to be enemies. By treating the safety shield not as a fixed wall but as a living guide that evolves with the agent's experience, it is possible to navigate dangerous environments without a perfect map beforehand. The system proved that an agent can learn to be both safe and skilled, provided the safety mechanism is flexible enough to grow as the agent's understanding of the world deepens. This approach offers a practical path forward for deploying intelligent systems in real-world scenarios where the rules are not fully known, ensuring that the journey to mastery does not come at the cost of safety.

Technical Summary: Adaptive Probabilistic Shielding by Learning MDPs for Safe Reinforcement Learning

Problem Statement

Safe Reinforcement Learning (RL) often relies on probabilistic shielding, a technique where a static "shield" constrains an agent's actions to ensure safety constraints are met. Traditionally, this shield is computed from a fully specified Markov Decision Process (MDP) model. However, in typical RL applications, the transition probabilities of the environment are unknown, rendering standard shielding inapplicable.

Existing approaches to this gap often involve a two-step process: first, collecting data (often unsafely) to learn a full MDP model, and then computing a shield. This fails in safety-critical real-world scenarios where safety must be maintained during the data collection phase.

This paper addresses the problem of Safe RL in settings where the MDP topology (state-action graph) is known, but the transition probabilities are unknown. The goal is to learn a policy that maximizes reward while adhering to safety constraints, without prior knowledge of the transition dynamics. Crucially, the authors acknowledge that while safety violations may not be entirely avoidable, the approach places great importance on minimizing them during the exploration and model learning phases, rather than guaranteeing zero violations.

Methodology

The authors propose a paradigm of Adaptive Probabilistic Shielding, which integrates model estimation, shield synthesis, and policy learning into a single online loop.

1. Core Architecture

The approach operates on an unknown MDP ( $M$ ) with a known underlying transition structure (an unknown MDP or uMDP, $M_U$ ). The system consists of three interacting components:

RL Agent: Uses Q-learning to learn a policy $\pi$ .
Model Estimator: Continuously updates an estimate of the transition probabilities based on observed transitions $(s, a, s')$ .
Adaptive Shield: A runtime monitor $\nabla$ that restricts the agent's actions based on the current model estimate.

2. Model Estimation and Interval MDPs (iMDPs)

Since exact probabilities are unknown, the system constructs Interval MDPs (iMDPs) from the collected data. An iMDP defines upper and lower bounds on transition probabilities rather than point estimates. The paper evaluates three specific estimators:

MAP (Maximum A-Posteriori): Produces point estimates with additive smoothing.
PAC (Probably Approximately Correct): Generates probabilistic intervals using Hoeffding's inequality to guarantee that the true MDP lies within the estimated intervals with high probability.
LUI (Linearly Updating Intervals): Iteratively updates probability intervals based on observed frequencies and prior strengths, without strict PAC guarantees but with convergence properties.

3. Shield Synthesis

Using the estimated iMDP, a shield is synthesized using the probabilistic model checker PRISM. The shield ensures that for a given horizon $h$ , the probability of reaching an unsafe state remains below a threshold $\theta$ .

Robust (Pessimistic) Shield: Assumes the worst-case transition probabilities within the intervals (minimizing safety probability). This is conservative but safe against estimation errors.
Optimistic Shield: Assumes the best-case transition probabilities (maximizing safety probability), allowing for more exploration.
Fallback Mechanism: If no action satisfies the strict safety threshold $\theta$ , the shield allows actions that are $\kappa$ -close to the safest available action, preventing the agent from being completely locked out of the environment.

4. The Adaptive Loop (Algorithm 1)

The algorithm proceeds in episodes. Every $u$ episodes (the update delay), the system:

Aggregates the transition database $D$ .
Re-estimates the iMDP using the chosen estimator.
Synthesizes a new shield based on the updated iMDP.
Continues training the RL agent under the new shield.

Exploration Strategy: To prevent the shield from stifling exploration (a "conservative trap" where the agent cannot gather data to refine the model), the authors extend the $\epsilon$ -greedy strategy. When the agent chooses to "explore" (with probability $\epsilon$ ), it selects a random action from the full action space $A$ , ignoring the shield's restrictions. This ensures that potentially safe but currently unverified actions are still tested, allowing the model to refine its estimates.

Key Contributions

Adaptive Shielding Paradigm: The paper introduces a method to compute and update probabilistic shields online as the MDP model is learned, addressing the limitation that traditional shielding requires a fully known model.
Integration of Estimation and Safety: It proposes a unified framework where model estimation, shield updates, and policy learning occur simultaneously, aiming to minimize safety violations even during the initial, high-uncertainty phases of learning.
Empirical Evaluation of Estimators: The authors provide a comprehensive comparison of MAP, PAC, and LUI estimators, analyzing how their specific properties (conservatism, convergence speed, data requirements) impact the trade-off between safety and reward.
Analysis of Design Choices: The paper systematically investigates critical hyperparameters, including the frequency of shield updates, the lookahead horizon ( $h$ ), and the impact of unshielded exploration on learning efficiency.

Experimental Results

The approach was evaluated across five environments: Aircraft, Antlion, Sinkholes, Crossroads, and Gravity. Key findings include:

Safety vs. Performance: The adaptive shielding approach achieves safety levels comparable to an "oracle" baseline (which uses the ground-truth MDP) and significantly outperforms unshielded RL in terms of safety violations, particularly in high-risk environments like Crossroads and Gravity.
Estimator Impact:
- Robust estimators (PAC/LUI) generally provide safer policies but can be overly conservative, sometimes preventing the discovery of optimal routes if the initial estimates are too wide.
- Optimistic estimators allow for more exploration and often achieve higher rewards but carry a higher risk of safety violations if the estimates are inaccurate.
- MAP estimators often failed to explore sufficiently, getting stuck in local optima due to the lack of uncertainty bounds.
Exploration Strategy: Allowing exploration outside the shield's current admissible set (UnifA) was crucial. Restricting exploration to only shield-admissible actions (Unif $\nabla(s)$ ) often prevented the agent from discovering better routes, particularly in the Antlion and Gravity environments.
Update Frequency: The frequency of shield updates ( $u$ ) significantly impacts performance. Too frequent updates can destabilize learning, while too infrequent updates delay the refinement of safety guarantees. No single update delay was optimal across all environments.
Lookahead Horizon ( $h$ ): Longer horizons generally improved safety in complex environments (like Gravity) but were redundant in short-horizon tasks (like Aircraft).

Significance and Claims

The paper claims that adaptive probabilistic shielding offers a practical solution for safe RL in scenarios where the environment model is partially known (topology only) and transition probabilities must be learned online.

The authors emphasize that their approach balances the exploration-exploitation dilemma with safety constraints. By allowing the shield to adapt as the model improves, the system avoids the "conservative trap" where a static, overly cautious shield prevents the agent from ever learning the true dynamics of the environment. The work demonstrates that it is possible to maintain safety guarantees and significantly reduce violations during the learning process itself, rather than just after the model is fully learned, making the technique applicable to real-world scenarios where safety violations are costly.

The paper remains modest in its claims, acknowledging that the approach relies on the assumption of a known state-action graph, that safety violations may not be entirely avoidable, and that the choice of estimator and hyperparameters requires careful tuning depending on the specific environment characteristics.