🤖 AI

Safe Exploration via Policy Priors

This paper introduces SOOPER, a safe reinforcement learning framework that leverages conservative policy priors and probabilistic dynamics models to guarantee safety during online exploration while achieving optimal convergence and outperforming state-of-the-art methods on both benchmarks and real-world hardware.

Original authors: Manuel Wendl, Yarden As, Manish Prajapat, Anton Pollak, Stelian Coros, Andreas Krause

Published 2026-08-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Manuel Wendl, Yarden As, Manish Prajapat, Anton Pollak, Stelian Coros, Andreas Krause

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine teaching a robot to do something new, like walking across a room or driving a race car. In the world of artificial intelligence, this is called "Reinforcement Learning." It's like teaching a dog a trick: the robot tries things, gets a "treat" (a reward) for doing well, and a "scolding" (a penalty) for doing poorly. Over time, it figures out the best way to behave. But here's the catch: in the real physical world, you can't let a robot just "try and fail" wildly. If a robot learning to walk falls over too hard, it might break its legs. If a self-driving car learns by crashing into walls, it could hurt people. This is the big problem of "Safe Exploration": how do you let an agent learn and get better without ever taking a step that could cause a disaster?

Scientists have been trying to solve this by giving robots a "safety net." Usually, this means having a backup plan or a strict rulebook that stops the robot before it gets hurt. But these safety nets can be so strict that the robot never learns anything new; it just stays safe but stuck. The challenge is finding a way to be brave enough to explore new, better ways of doing things, while being smart enough to know exactly when to pull back and stay safe. This paper tackles that exact dilemma, proposing a new way for robots to learn online, in real-time, without ever crossing the line into danger.

The paper introduces a new algorithm called SOOPER (which stands for Safe Online Optimism for Pessimistic Expansion in RL). Think of SOOPER as a robot with a very cautious, experienced mentor. This mentor is a "policy prior"—a set of instructions the robot already knows, perhaps learned from a simulator or old data. This mentor is "pessimistic," meaning it assumes the worst-case scenario and only does the bare minimum to stay safe. It's like a parent who knows exactly how to walk across a slippery floor without falling, but isn't very fast or stylish.

SOOPER's clever trick is to let the robot be "optimistic" while it's learning. The robot is allowed to try new, risky moves to see if it can find a faster or more efficient path. However, it has a built-in safety switch. As the robot tries these new moves, it constantly calculates: "If I keep going this way, will I eventually run out of safety budget?" If the answer is even a little bit "maybe," the robot instantly switches to its pessimistic mentor. The mentor takes over, guiding the robot safely back to a secure state. This switch happens so quickly that the robot never actually gets into trouble.

Here is the magic part: the robot doesn't just stop and wait. When the mentor takes over, the robot records that moment as a "lesson learned." It realizes, "Oh, that path I was trying leads to a dead end where I have to be slow and careful." This information helps the robot build a better mental map of the world. Over time, the robot learns exactly where the safe boundaries are and discovers new, faster routes that stay within those boundaries. It's like a hiker exploring a forest with a guide who knows the safe trails. The hiker tries to cut through the bushes to find a shortcut. If they get too close to a cliff, the guide grabs their hand and pulls them back to the safe path. The hiker learns, "Okay, that shortcut was too dangerous," and next time, they try a different route. Eventually, the hiker finds a shortcut that is both safe and fast, without ever falling off a cliff.

The authors prove mathematically that this method guarantees safety at every single step of the learning process. They also show that the robot's performance improves over time, eventually finding a strategy that is nearly as good as the best possible strategy, all while never violating safety rules. They tested this on computer simulations of various tasks, like swinging a pole up to a vertical position and navigating a race car around obstacles. In every case, SOOPER stayed safe while learning faster and performing better than other top methods.

Most impressively, they didn't just stop at computer simulations. They took SOOPER and put it on a real, physical remote-controlled race car. This car has to drive at high speeds, dodging tires and reaching a goal. The real world is messy and unpredictable, with delays in the car's motors and sensors. Even with these real-world glitches, SOOPER successfully learned to drive faster and more efficiently than the initial "safe" driving style, all while never crashing into the obstacles. The paper suggests that this approach could be a major step forward for getting robots to learn safely in the real world, moving them from controlled labs to our actual streets and homes.

Technical Summary: SOOPER - Safe Exploration via Policy Priors

Problem Statement

Reinforcement learning (RL) agents require the ability to learn and adapt online in physical environments, but they face a critical challenge: safe exploration. Agents must discover optimal behaviors without taking actions that lead to catastrophic outcomes, even when they lack complete knowledge of their environment. While theoretical methods often provide safety guarantees, they struggle to scale to complex, high-dimensional tasks. Conversely, scalable deep RL approaches often fail to ensure safety during the learning process.

A key limitation in existing approaches is how they incorporate prior knowledge. Most methods use priors primarily to maintain safety (e.g., via backup policies) but fail to leverage these priors to actively guide exploration toward promising regions of the state space. This paper addresses the need for an algorithm that can utilize suboptimal yet conservative policies (derived from offline data or simulators) as priors to guarantee safety while simultaneously enabling efficient, near-optimal learning.

Methodology: SOOPER

The authors propose SOOPER (Safe Online Optimism for Pessimistic Expansion in RL), a model-based RL algorithm designed for Constrained Markov Decision Processes (CMDPs) in continuous state-action spaces. The core philosophy is to use pessimistic policy priors to ensure safety and optimistic planning to drive exploration.

1. Core Mechanisms

Pessimistic Policy Priors: SOOPER assumes access to a conservative policy prior $\hat{\pi}$ (e.g., trained on offline data or a simulator with distribution shifts). This prior is guaranteed to satisfy safety constraints under a set of plausible dynamics models $\mathcal{F}_0$ .
Online Cost Tracking (Safety Guarantee): During real-world rollouts, SOOPER monitors the accumulated cost. It employs a switching mechanism:
- If the current policy $\pi_n$ appears safe based on a pessimistic cost value function $Q^{\hat{\pi}}_{c,n}$ (which includes an uncertainty penalty), the agent executes $\pi_n$ .
- If the predicted cost exceeds the safety budget $d$ , the agent immediately switches to the safe prior $\hat{\pi}$ .
- This ensures that the agent never violates constraints with high probability, even when the learned dynamics model is inaccurate.
Simulated Exploration (Optimism): To learn the environment dynamics without risking safety, SOOPER constructs a "planning MDP" ( $\tilde{M}$ $\tilde{M}$ ) based on the learned world model.
- Termination: In this simulated environment, trajectories that would trigger the safe prior $\hat{\pi}$ in the real world are terminated.
- Terminal Reward: Upon termination, the agent receives a pessimistic reward value $V^{\hat{\pi}}_r$ (the worst-case value of the prior policy). This incentivizes the agent to find trajectories that outperform the prior without triggering the safety switch.
- Intrinsic Rewards: The objective function includes intrinsic rewards based on model uncertainty ( $\|\sigma_n\|$ ) to encourage both exploration (gathering data to reduce uncertainty) and expansion (proactively enlarging the set of known safe policies).

2. Theoretical Framework

Regret Analysis: Unlike prior works that only guarantee optimality at the end of training (simple regret), SOOPER establishes a bound on cumulative regret. This guarantees that the agent's performance remains high throughout the learning process, not just at convergence.
Convergence: The algorithm is proven to converge to an optimal policy by iteratively refining the dynamics model and expanding the implicit set of safe policies. The cumulative regret is shown to grow sub-linearly with the number of episodes $N$ .

Key Contributions

Algorithm Proposal: Introduction of SOOPER, a scalable, model-based RL algorithm that leverages offline or simulation-trained policies as safe priors for continuous control tasks.
Theoretical Guarantees:
- Safety: Proof that SOOPER satisfies safety constraints with high probability throughout all learning episodes.
- Regret Bound: Establishment of a novel upper bound on cumulative regret, ensuring good performance during learning rather than just at the final iteration.
- Expansion: Demonstration that the algorithm implicitly expands the set of safe policies until it converges to the feasible set defined by the true CMDP.
Empirical Validation: Extensive experiments on standard safe RL benchmarks (RWRL, SafetyGym) and real-world hardware (a remote-controlled race car).

Experimental Results

The authors evaluated SOOPER against state-of-the-art baselines, including SAILR, CRPO, and Primal-Dual optimization methods.

Sim-to-Real and Distribution Shifts: In tasks involving dynamics mismatches (e.g., varying gear parameters, pole lengths), SOOPER consistently satisfied safety constraints while outperforming or matching baselines in terms of cumulative reward.
Vision Control: The method successfully scaled to tasks with image-based observations (CartpoleSwingup), learning dynamics directly from embeddings of a pretrained vision encoder.
Offline-to-Online: When initialized with policies trained on large offline datasets, SOOPER improved performance significantly while maintaining safety.
Real-World Hardware: In experiments with a high-frequency (60 Hz) remote-controlled race car, SOOPER learned to navigate around obstacles and reach goals. It achieved roughly twice the reward of the initial prior policy while strictly adhering to safety constraints throughout training.
Ablation Studies: The authors demonstrated that:
- Terminations in the simulated MDP are crucial for reducing dependency on the prior.
- The method is robust to varying levels of conservatism in the prior policy.
- The approach remains safe under non-Gaussian, non-additive noise disturbances.

Significance and Claims

The paper claims that SOOPER represents a significant step toward deployable RL by bridging the gap between theoretical safety guarantees and practical scalability.

Unified Objective: By reformulating the safety problem as an unconstrained MDP with specific termination and reward structures, SOOPER bypasses the complexity of directly solving CMDPs, making it compatible with standard deep RL architectures (e.g., MBPO).
Performance During Learning: The authors emphasize that their cumulative regret bound is a distinct improvement over prior works (like SAILR or ActSafe) that only guarantee safety and optimality at the end of training, often at the cost of arbitrarily poor performance during the exploration phase.
Practicality: The method is shown to be computationally efficient, with runtime comparable to standard model-based RL baselines, and effective in real-world scenarios with stochasticity and delays.

The authors conclude that while challenges remain (such as handling non-episodic settings or high-probability constraints on specific states), SOOPER provides a robust framework for safe online learning that translates theoretical guarantees into real-world performance.