How Safe Will I Be Given What I Saw? Calibrated Prediction of Safety Chances for Image-Controlled Autonomy

1. Problem Statement

The paper addresses the critical challenge of safety prediction for autonomous robots that rely on end-to-end vision-controlled systems (e.g., deep neural networks mapping raw images directly to actions).

• The Core Difficulty: Traditional safety verification methods often require access to low-dimensional state models (e.g., position, velocity) and explicit dynamical equations. However, modern image-controlled systems operate in a "black-box" manner where the state-transition model ($f$) and observation model ($o$) are unknown or inaccessible.
• The Specific Challenge: Predicting safety over long horizons (future time steps) from high-dimensional image sequences is prone to distribution shift. As a predictor forecasts future frames, small errors compound, causing the predicted visual trajectory to deviate significantly from the training distribution. This leads to miscalibrated risk estimates (overconfidence or underconfidence) and high false-positive rates in safety evaluators.
• Goal: To develop a framework that predicts the probability of future safety ($P(\phi(x_{i+k}) | y_i)$) directly from image sequences and provides statistically calibrated confidence bounds on these predictions without requiring manual relabeling during deployment.

2. Methodology

The authors propose a modular framework comprising four key components:

A. Modular Prediction Pipelines

The paper introduces two architectural paradigms for safety prediction:

1. Monolithic Predictors: A single network maps an image sequence directly to a safety probability. While simple, these struggle to learn coherent long-term dynamics.
2. Composite Predictors (World Model-based): These decompose the task into three stages:
   • Encoder: Compresses raw images into a compact latent space using a Variational Autoencoder (VAE) or Vector Quantized VAE (VQ-VAE).
   • Forecaster: Predicts future latent states (trajectories) using Recurrent Neural Networks (LSTM, BiLSTM) or Transformers.
   • Evaluator: Assesses the safety of the predicted latent trajectory.
   • Key Insight: The composite approach explicitly separates dynamics learning from safety evaluation, reducing the complexity of the learning task.

B. Unsupervised Domain Adaptation (UDA)

To combat distribution shift in long-horizon predictions (where predicted images/latents differ from training data):

• The authors employ Marginal Entropy Minimization (MEMO) at test time.
• Mechanism: For a given input, the system generates multiple stochastically augmented views (e.g., rotation, noise). The safety evaluator is fine-tuned via a single gradient descent step to minimize the entropy of its predictions across these views.
• Benefit: This forces the evaluator to produce confident, consistent predictions on shifted data without requiring new ground-truth labels.

C. Conformal Calibration

To ensure the predicted probabilities are trustworthy:

• The authors use Adaptive Binning combined with Conformal Prediction.
• Instead of calibrating individual binary labels, they calibrate the average safety probability within bins of data.
• Algorithm: They compute non-conformity scores (absolute difference between predicted and true average safety) on a validation set. Using these scores, they derive a statistical upper bound ($c$) such that the true safety probability lies within $[P' - c, P' + c]$ with a guaranteed confidence level ($1-\alpha$).

D. Training Strategy

• Two-Stage Training: First, the VAE is trained to reconstruct images. Second, the latent forecasters and evaluators are trained on the latent representations.
• Loss Functions: VAEs use reconstruction + KL divergence loss; Forecasters use Mean Squared Error (MSE); Evaluators use Cross-Entropy.
• Data Handling: The dataset is structured as independent trajectories to satisfy the i.i.d. assumption required for conformal prediction.

3. Key Contributions

1. Modular Pipeline Family: A flexible set of safety predictors for image-controlled robots, comparing monolithic vs. composite (world model) approaches.
2. Robustness via UDA: A novel application of unsupervised domain adaptation (MEMO) to safety evaluators, enabling them to maintain accuracy under distribution shifts caused by compounding prediction errors.
3. Theoretical Calibration Guarantees: The first instantiation of conformal calibration for safety chance predictions in a model-free, end-to-end vision setting, providing rigorous statistical bounds on prediction error.
4. Comprehensive Evaluation: Extensive experiments on three benchmarks (Racing Car, Cart Pole, and physical Donkey Cars) demonstrating the efficacy of the approach.

4. Experimental Results

The framework was evaluated on three benchmarks with varying prediction horizons ($k$):

• Composite vs. Monolithic: Composite predictors (World Models) significantly outperformed monolithic ones, especially on long horizons.
  • Reasoning: Monolithic models fail to capture long-term temporal dependencies, whereas composite models explicitly model dynamics before evaluating safety.
  • Result: Composite models achieved higher F1 scores and significantly lower False Positive Rates (FPR).
• Architecture Impact: Transformers and VQ-VAEs (discrete latent spaces) generally outperformed standard LSTMs and continuous VAEs, particularly in calibration metrics (lower ECE and Brier scores).
• UDA Effectiveness: Evaluators equipped with UDA showed remarkable robustness to Out-of-Distribution (OOD) data.
  • Example: On the Racing Car OOD test set, the UDA Image Evaluator achieved an F1 score of 81.65% compared to 63.94% for the standard Image Evaluator, while maintaining performance on normal data.
• Calibration Reliability:
  • The conformal calibration successfully reduced Expected Calibration Error (ECE) by an order of magnitude (e.g., from ~0.13 to ~0.01).
  • The predicted confidence intervals contained the true safety probability in >95% of cases across all benchmarks, validating the theoretical guarantees.

5. Significance and Impact

• Bridging the Gap: This work bridges the gap between high-performance end-to-end vision control and rigorous safety assurance, a critical step for deploying autonomous robots in unstructured environments.
• Trustworthy Autonomy: By providing calibrated probabilities rather than just binary labels, the system allows operators to understand the uncertainty of a prediction. This is vital for safety-critical decisions where false negatives (predicting safe when unsafe) are catastrophic.
• Scalability: The method does not require low-dimensional state models or manual labeling during deployment, making it applicable to real-world systems where such data is unavailable.
• Future Direction: The paper highlights the potential of integrating large foundation models and vision-language models to further improve zero-shot generalization and safety reasoning.

In summary, the paper presents a robust, theoretically grounded framework that enables autonomous robots to predict their own safety risks from raw camera inputs with high accuracy and statistically guaranteed confidence, even when facing novel environmental conditions.