Local Conformal Calibration of Dynamics Uncertainty from Semantic Images

Technical Summary: Local Conformal Calibration of Dynamics Uncertainty from Semantic Images

Problem Statement
Robotic systems operating in real-world environments face significant challenges in uncertainty quantification due to unmodeled disturbances (aleatoric uncertainty) and model mismatch (epistemic uncertainty). While Conformal Prediction (CP) offers a distribution-free framework for providing probabilistic guarantees on prediction regions, existing CP methods in robotics suffer from two critical limitations:

1. Lack of Discriminability: Standard CP provides average (marginal) coverage guarantees over a dataset but fails to distinguish between state-action regions with high versus low model mismatch. This prevents planners from steering the system toward lower-uncertainty states.
2. Environment-Specific Data Requirement: Current approaches typically require calibration data collected from the exact test-time environment to ensure validity. This is impractical and risky, as it necessitates traversing new, potentially hazardous environments (e.g., icy roads) solely to calibrate the dynamics model.

The authors address the need for an algorithm that can provide provable, finite-sample uncertainty guarantees for unseen test-time environments by leveraging perception information from visually similar, but distinct, environments.

Methodology: OCULAR
The paper introduces Observation-aware Conformal Uncertainty Local-Calibration (OCULAR), a method that calibrates a linear Gaussian dynamics model ($\tilde{f}$) using robot-frame perception data. The core innovation is replacing world-frame poses (which require environment-specific data) with a learned representation of semantic observations.

The methodology proceeds in two phases:

1. Offline Calibration:

   • Observation Processing: Raw depth and semantic images are projected into a robot-frame "planar semantic footprint" ($o'_t$). This representation captures the environment ahead of the robot while discarding irrelevant world-frame coordinates.
   • Latent Encoding: A Convolutional Autoencoder (CAE) is trained on data from a set of calibration environments to compress the planar footprints into a low-dimensional latent vector ($\phi_t$).
   • Local Partitioning: The input space is defined as $X = (v_t, a_t, \phi_t)$, combining velocity, action, and the latent observation. A Decision Tree (DTree) is trained to partition this space into regions ($X_k$) where the nonconformity scores (disagreement between $\tilde{f}$ and ground truth) are approximately constant. This is achieved using the LOCART algorithm to minimize score variance within partitions.
   • Local Calibration: For each partition $X_k$, a Split Conformal Prediction (SplitCP) procedure is performed on a hold-out subset of the calibration data. This yields a partition-specific scaling factor ($\hat{q}_k$) that adjusts the covariance of the Gaussian predictions.

2. Online Deployment:

   • During operation in a new, unseen environment, the robot processes its current observation into the latent space and passes it through the pre-trained DTree to identify the current partition $X_k$.
   • The dynamics model's predicted covariance is scaled by the corresponding factor $\xi_k$ derived from $\hat{q}_k$.
   • This calibrated uncertainty estimate is fed into a Model Predictive Control (MPC) planner. The planner constructs a prediction region that is guaranteed to contain the true next state with probability $(1-\alpha)$ and optimizes trajectories to avoid regions where the calibrated uncertainty is high or intersects with obstacles.

Key Contributions
The authors make three primary contributions:

1. Algorithm Proposal: They propose OCULAR, an algorithm that provides provable finite-sample guarantees on dynamics uncertainty in a new, unseen environment using only data from visually similar environments.
2. Theoretical Validity: They prove that the pipeline of projecting observations, encoding them via a CAE, and partitioning via a Decision Tree preserves the exchangeability required for Conformal Prediction. This allows for local (conditional) coverage guarantees rather than just global marginal guarantees.
3. Empirical Validation: They validate the approach on a double-integrator system subject to random perturbations and significant model mismatch (simulating slippery surfaces). Experiments utilize both simplified planar sensors and realistic Isaac Sim cameras.

Results
Experiments were conducted on four planar maps and three Isaac Sim road scenarios, comparing OCULAR against:

• NoCP: Uncalibrated model (fails on out-of-distribution (OOD) data).
• SplitCP: Global calibration using test-environment data (over-conservative in-distribution, under-protective OOD).
• LUCCa: Local calibration using test-environment data (requires environment-specific data).

Key findings include:

• Coverage: OCULAR achieves the user-set coverage probability (90%) in both in-distribution and OOD regions (e.g., icy roads) without using any data from the test environment. In contrast, NoCP severely undercovers OOD regions, and SplitCP fails to adapt to local uncertainty variations.
• Volume Efficiency: OCULAR constructs prediction regions that are only 12–15% larger than an "oracle" optimal region, demonstrating high volume efficiency comparable to methods that have access to test-environment data (like LUCCa).
• Safe Planning: In trajectory optimization tasks, OCULAR successfully navigates high-uncertainty regions by slowing down and avoiding collisions, achieving 100% success rates in all tested maps. Methods without calibration (NoCP) or with global calibration (SplitCP) frequently resulted in collisions due to over-optimism or excessive conservatism, respectively.

Significance and Claims
The paper claims that OCULAR bridges the gap between theoretical uncertainty guarantees and practical robotic deployment. By utilizing perception information to generalize across environments, the method eliminates the need for risky, environment-specific calibration data collection. The authors emphasize that their guarantees are non-asymptotic and distribution-free, requiring no strong assumptions about the unknown real system dynamics.

The authors modestly note that while they demonstrate one-step safety guarantees and heuristic multi-step planning, they do not claim rigorous theoretical guarantees for multi-step safety due to the non-exchangeability of map-generated observations during execution. Furthermore, they acknowledge sensitivity to hyperparameters in the CAE and Decision Tree, and that the current latent representation learning is decoupled from the uncertainty prediction utility, suggesting an avenue for future work.