Technical Summary: Conformal Reachability for Safe Control in Unknown Environments

1. Problem Statement

The paper addresses the challenge of designing provably safe control policies for autonomous systems operating in unknown dynamical environments. While Reinforcement Learning (RL) has advanced autonomy, standard approaches often lack rigorous safety guarantees, typically optimizing for average performance or relying on empirical safety that fails in safety-critical scenarios.

Existing verification and synthesis techniques face significant limitations:

• Known/Deterministic Assumptions: Many methods assume system dynamics are known or deterministic, which is rarely true in practice.
• Finite State/Action Spaces: Approaches often require finite state or action spaces, limiting applicability to continuous systems.
• Scalability and Smoothness: Formal methods like Hamilton-Jacobi reachability do not scale well with system complexity, and many learning-based methods assume smooth dynamics or specific model forms (e.g., Gaussian Processes).
• Approximate Guarantees: Current safe RL methods often provide safety only "on average" (e.g., via Constrained MDPs) or in the limit, rather than offering finite-horizon probabilistic guarantees for unknown, potentially stochastic dynamics.

The core problem is to learn a policy that maximizes cumulative reward while ensuring that the system remains within a safe set $S_s$ for a decision horizon $K$ with a rigorous, high-probability guarantee, despite having no prior knowledge of the system dynamics $P$ or the reward function $r$.

2. Methodology: ReCORS

The authors propose ReCORS (Reachability with COnformal uncertainty sets for Reinforcement learning with Safety guarantees), a framework that integrates model-based and model-free RL with Conformal Prediction (CP) and reachability analysis.

2.1 System Dynamics Approximation

The method learns a parametric, discrete-time dynamics model $\hat{f}_\beta$ (a neural network) that maps state-action pairs to predicted next states.

• Safety-Aware Training: Unlike standard dynamics learning, the loss function for $\hat{f}_\beta$ incorporates safety weights. Dimensions of the state space that are more critical to safety constraints (e.g., proximity to obstacles) are assigned higher weights in the $\ell_2$ loss. This prioritizes prediction accuracy in safety-critical dimensions, even if it slightly increases overall prediction error.
• Training Pipeline: The model is pre-trained on random actions and then fine-tuned iteratively alongside the policy using data collected from the current policy.

2.2 Conformal Safety Analysis (CSA)

To provide probabilistic guarantees without assuming specific dynamics distributions, ReCORS employs Conformal Prediction.

• Uncertainty Quantification: CP is used to generate valid prediction intervals (uncertainty sets) for the dynamics model. For a given state and action, the true next state is guaranteed to lie within the interval $[\hat{f}(s,a) - \eta, \hat{f}(s,a) + \eta]$ with probability $1-\alpha$.
• Trajectory-Level Bounds: To avoid overly conservative union bounds over time steps, the authors utilize a trajectory-level nonconformity score (based on Cleaveland et al. [12]). This optimizes weights for each time step to produce tighter bounds over the entire horizon $K$.
• Reachability Verification: Using the learned dynamics $\hat{f}_\beta$ and the conformal uncertainty bounds $\eta$, the system performs forward reachability analysis. It verifies whether the reachable set of states at any time $t \in [1, K]$ intersects with the unsafe set $S_u$.
• Initial State Distribution: The framework accounts for uncertainty in the initial state distribution $\rho$ by applying finite-sample tail bounds (Hoeffding's inequality) to the empirical verification results, ensuring the safety guarantee holds for the entire distribution, not just the sampled initial states.

2.3 Differentiable Training Pipeline

A key innovation is making the safety verification process differentiable to enable end-to-end policy optimization.

• Neural Conformal Predictor: A separate neural network $\eta_\omega$ is trained to predict the conformity thresholds $\eta$. This replaces the non-differentiable quantile calculation in standard CP.
• Loss Functions:
  • Conformal Loss: Combines an efficiency term (minimizing the size of the prediction set) and a coverage term (ensuring empirical coverage meets the target $1-\alpha$).
  • Safety Loss: Explicitly penalizes the maximum safety violation over the horizon and encourages the improvement of safety scores from the initial to the final step.
• Joint Optimization: The policy (actor), critic, dynamics model, and conformal predictor are optimized jointly. The total loss combines the standard RL reward objective with the conformal and safety losses.
• Curriculum Learning: The algorithm employs a curriculum strategy, starting with a short safety horizon $K_0$ and incrementally increasing it as the policy achieves provable safety, allowing the system to learn complex behaviors gradually.

3. Key Contributions

1. Conformal Safety Analysis (CSA): A novel framework combining conformal prediction with reachability analysis for learning-based controllers in unknown dynamical systems. It provides rigorous probabilistic safety guarantees without assuming smoothness or specific forms of the underlying dynamics.
2. Provable Safe Policy Learning: A training framework that integrates model-free RL (for reward maximization) with model-based reachability estimation (for safety). It enforces hard safety constraints via differentiable loss functions derived from conformal uncertainty sets.
3. Comprehensive Evaluation: Extensive experiments across seven environments (including Cartpole, Lane Following, 2D/3D Quadrotors, CarGoal, and HalfCheetah) with both linear (affine) and nonlinear safety specifications.

4. Experimental Results

The authors evaluated ReCORS against five state-of-the-art baselines (PPOLag, PCPO, CRPO, P3O, RESPO).

• Verified Safety: ReCORS achieved significantly stronger provable safety guarantees than all baselines. For instance, in the Cartpole environment, baselines failed to provide useful guarantees beyond a horizon of $K=20$, whereas ReCORS maintained high-probability safety guarantees up to $K \ge 50$.
• Reward vs. Safety Trade-off: While some baselines achieved higher rewards, they did so at the cost of poor safety properties (frequent violations). ReCORS was the only method to consistently achieve both strong performance and verified safety. In some domains (e.g., 2D Quadrotor), ReCORS outperformed all baselines in both reward and safety.
• Nonlinear Constraints: The approach successfully handled nonlinear safety constraints (e.g., safe distance in 2D Quadrotor), outperforming baselines that either failed to verify safety or resulted in conservative behaviors (e.g., drones staying stationary).
• Ablation Studies: Results confirmed the importance of the safety-aware weights in dynamics training and the joint optimization of the dynamics and uncertainty models.

5. Significance and Claims

The paper claims to present a novel approach that bridges the gap between high-performance learning and rigorous safety verification in unknown environments. Its significance lies in:

• Removing Restrictive Assumptions: Unlike prior work, it does not assume known, deterministic, or smooth dynamics, nor does it require finite state spaces.
• Distribution-Free Guarantees: By leveraging conformal prediction, it provides statistically calibrated, distribution-free uncertainty quantification, making the safety guarantees robust to model errors and unknown stochasticity.
• Scalability: The discrete-time finite-step reachability formulation scales better with system complexity compared to PDE-based Hamilton-Jacobi methods.
• Practical Applicability: The method is demonstrated to be effective across diverse domains, offering a viable path toward trustworthy autonomy where safety cannot be compromised for performance.

The authors acknowledge limitations, noting that the current approach assumes perfect observability of the system state, which remains an open problem for future work.