Approximating Safety Feedback Without a Safety Oracle via Model Predictive Control

Technical Summary: Approximating Safety Feedback Without a Safety Oracle via Model Predictive Control

Problem Statement
Safe decision-making for mobile robots often relies on the existence of a "safety oracle"—a mechanism to verify if proposed actions violate safety constraints. Current methods, such as Control Barrier Functions, Backward Reachable Tubes, and Safe Reinforcement Learning (Safe RL) discriminators, typically require prior knowledge of explicit safety constraints or access to ground-truth safety labels during training. However, in complex environments (e.g., high-speed navigation on uneven terrain or robotic manipulation in open work cells), safety constraints are often implicit, difficult to model mathematically, or incompletely understood. While simulators can model complex dynamics, they do not inherently provide a safety function. Existing approaches that attempt to learn safety proxies (e.g., via Gaussian Processes or deep learning discriminators) often require observable ground truth for labeling or make strong assumptions about the smoothness of the safety function, limiting their real-world applicability. Furthermore, many reversible exploration methods still risk taking unsafe actions during the learning process.

Methodology: SAVMPC
The paper proposes Safety Assessment Via Model-Predictive Control (SAVMPC), a filtering algorithm designed to approximate safety feedback without direct access to a safety oracle or hand-designed constraints. The method operates under two primary assumptions:

1. Positive Invariance of Unsafe States: The unsafe state space ($X_{unsafe}$) is positive invariant, meaning once an agent enters an unsafe state, it cannot recover to a safe state.
2. Reversibility as a Proxy: If an agent can transition from a current state $x$ to a future state $x'$ and subsequently find a sequence of actions to return from $x'$ back to $x$, then $x$ and $x'$ belong to the same set. Given the positive invariance of $X_{unsafe}$, if a return path exists to a known safe state, the intermediate states are also safe.

Algorithmic Workflow:
SAVMPC functions as a shield during action selection:

1. Action Proposal: The control policy $\pi$ proposes an action $u$ at state $x$.
2. Forward Projection: The action is projected forward using a simulator (black-box dynamics $f_{dyn}$) to a candidate next state $x'$.
3. Reversibility Check: The algorithm employs the Model-Predictive Path Integral (MPPI) controller to attempt to find a sequence of actions that transitions the agent from $x'$ back to the original state $x$ (or a previous state in the trajectory).
   • This is formulated as a feasibility problem: find a control sequence $u_0, \dots, u_{T-1}$ such that the resulting trajectory reaches $x$ within a tolerance $\delta$ over a time horizon $T$.
4. Decision:
   • If MPPI finds a valid return path, the action $u$ is deemed safe and executed.
   • If no return path is found within the sampling limit, the action is rejected, and the policy resamples. If resampling fails after $N$ attempts, the trajectory is aborted, and the environment is reset.

This approach allows the system to verify safety by ensuring that no action leads to a "dead end" from which the agent cannot return to a known safe region.

Key Contributions

1. SAVMPC Algorithm: A novel filtering algorithm that enables safe action selection by planning backward to known safe states using reversibility, eliminating the need for hand-constructed constraints or ground-truth safety labels.
2. Conservative Constraint Estimation: An extension of the reversibility planning algorithm that explicitly calculates a conservative estimate of unknown environment constraints.
3. Empirical Validation: Evaluation of SAVMPC as a shield in Reinforcement Learning (RL) training, demonstrating performance comparable to oracle-based shields while outperforming other reversibility-based approaches (RAC, LNT) in terms of safety guarantees.

Experimental Results
The authors evaluated SAVMPC in two environments: Continuous Cartpole and Two-Dimensional Navigation.

• Disagreement Study: In a simulation with 300,000 trajectories, SAVMPC was compared against a ground-truth safety oracle. The results showed that SAVMPC produces zero false negatives (it never classifies an unsafe state as safe). Disagreements occurred primarily near constraint boundaries, where the algorithm was more conservative (over-approximating the unsafe region) when using tighter safety tolerances ($\delta$).
• RL Training Performance:
  • Continuous Cartpole: SAVMPC achieved a final reward comparable to the PPO algorithm equipped with a true Safety Oracle shield and significantly higher than baseline PPO, RAC, and Leave No Trace (LNT). Crucially, SAVMPC maintained 0.0% constraint violations, whereas baseline and other reversible methods incurred significant violations (up to 77.3%).
  • 2D Navigation: SAVMPC outperformed the Safety Oracle shield in final reward (842.74 vs. 782.23) while maintaining 0.0% constraint violations. It converged faster than other methods, attributed to its conservative approximation shrinking the exploration space effectively.
  • Comparison: Unlike RAC and LNT, which learned safety approximations over time and still incurred violations, SAVMPC enforced safety from the first step without privileged information.

Significance and Claims
The paper claims that SAVMPC offers a practical solution for safe decision-making in environments where safety constraints are complex or unknown. By leveraging the assumption of positive invariance and the computational power of MPPI within a simulator, the method approximates the performance of a safety oracle without requiring ground-truth labels. The authors emphasize that the algorithm ensures an agent takes zero unsafe actions in the underlying environment, effectively bridging the gap between theoretical safety guarantees and the practical limitations of modeling complex physical constraints. The work suggests that reversibility can serve as a robust, data-free proxy for safety in scenarios where traditional constraint modeling is infeasible.