🤖 machine learning

Probabilistic Object Detection with Conformal Prediction

This paper addresses the challenges of applying Conformal Prediction to multi-output object detection by introducing a scaled, coordinate-wise approach with class-wise calibration and a two-step pipeline, which significantly improves prediction interval sharpness and IoU across autonomous driving datasets while maintaining rigorous coverage guarantees.

Original authors: Christopher Ries, Moussa Kassem Sbeyti, Nicolas Bianco, Nadja Klein

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Christopher Ries, Moussa Kassem Sbeyti, Nicolas Bianco, Nadja Klein

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are driving a self-driving car. The car's computer needs to spot other vehicles, pedestrians, and obstacles. It doesn't just need to say, "There's a car there"; it needs to know how sure it is about that car's exact location. If the car is wrong but acts like it's 100% sure, that's dangerous. If it's right but acts like it's guessing, it might brake unnecessarily.

This paper tackles the problem of making those "guesses" reliable and efficient. Here is the breakdown using simple analogies.

The Problem: The "One-Size-Fits-All" Safety Net

Standard AI models for object detection are like a factory that makes safety nets. They try to catch the true location of an object (like a pedestrian) within a box.

The Old Way (Unscaled Conformal Prediction): Imagine the factory makes every single safety net the exact same size, no matter what.
- If the object is easy to see (a clear day, a big truck), the net is way too big. It wastes space and makes the car think the truck could be anywhere in a huge area.
- If the object is hard to see (fog, a tiny bicycle), the net might actually be too small, and the car might miss the danger.
- The Result: The system is safe (it catches the object), but it's clumsy and inefficient.

The Solution: The "Smart, Stretchy" Safety Net

The authors propose a new method called Scaled Conformal Prediction. Think of this as a safety net made of smart, stretchy material.

How it works: The system uses a special "uncertainty meter" (derived from a technique called loss attenuation) to measure how hard a specific prediction is.
- Easy prediction? The meter says, "This is easy!" The net stays small and tight around the object.
- Hard prediction? The meter says, "This is tricky!" The net stretches out wider to ensure it still catches the object.
The Benefit: The car gets a precise location when things are clear and a wide safety zone when things are foggy, without ever losing its guarantee of safety.

The "Two-Step" Dance

Object detection has two jobs:

What is it? (Is it a car or a dog?)
Where is it? (What are the coordinates?)

The paper tests a "two-step pipeline" (inspired by a method called RAPS):

First, the system makes a shortlist of possible classes (e.g., "It's probably a car, maybe a truck").
Then, it draws the safety net based only on that shortlist.

The authors found that when you combine this "shortlist" method with their "smart, stretchy" net, the system works even better. The "smart" net adapts to the difficulty of the specific object, whereas the "old" fixed-size net just gets bigger and clumsier when you add the shortlist step.

The Results: Sharper, Safer, and Smarter

The team tested this on three different driving datasets (KITTI, BDD, and CODA), including a scenario where the car was tested in a completely different environment than it was trained on (like driving in a new city with different weather).

Accuracy: The "smart, stretchy" nets were much tighter to the actual objects. In some cases, they improved the alignment (called IoU) by up to 19% compared to the old fixed-size nets.
Efficiency: The "score" for how well the net balanced size and safety improved by up to 39%.
Safety: Crucially, making the nets tighter did not make them less safe. They still caught the objects at the guaranteed rate (e.g., 90% of the time).
Robustness: Even when the car was driving in a new, unfamiliar environment (distribution shift), the smart nets still outperformed the old ones.

The "Calibration" Twist

The authors also checked if they needed to "calibrate" the uncertainty meter (like tuning a radio to get a clear signal).

They found that the raw, uncalibrated meter was already 90% of the way to perfect.
Fine-tuning it (calibration) helped a little bit, but the biggest win was simply switching from the "fixed-size" net to the "stretchy" net in the first place.

Summary

The paper proves that for self-driving cars, you don't need to guess the same way for every situation. By using a method that adapts the size of the "safety box" based on how difficult the current view is, you get a system that is sharper (knows exactly where things are), safer (still guarantees it won't miss them), and more efficient (doesn't waste space on easy tasks).

Where to find the code: The authors have made their tools available at https://github.com/mos-ks/OD-CP.

Technical Summary: Probabilistic Object Detection with Conformal Prediction

Problem Statement
Deploying object detectors in safety-critical domains like autonomous driving requires not only accurate predictions but also reliable uncertainty estimates. State-of-the-art detectors typically rely on deterministic neural networks that lack inherent uncertainty quantification (UQ). While probabilistic detectors exist, their UQ methods are often model-specific and computationally expensive. Conformal Prediction (CP) offers a post-hoc, model-agnostic, and distribution-free framework to construct prediction sets with finite-sample marginal coverage guarantees. However, applying CP to object detection presents two specific challenges:

Structured Multi-Output: Object detection produces multi-output predictions (bounding box coordinates and class labels), complicating the application of classical CP theory designed for single outputs.
Fixed-Width Intervals: Standard (unscaled) CP generates prediction intervals of constant width across all inputs. This ignores the varying difficulty of individual predictions, leading to unnecessarily wide intervals for low-uncertainty cases and potentially insufficient coverage for high-uncertainty cases.

Methodology
The authors propose a framework that integrates Conformal Prediction with a probabilistic object detector trained using loss attenuation (Kendall and Gal, 2017; Kassem Sbeyti et al., 2023). This detector learns per-prediction aleatoric uncertainty (variance) alongside regression targets, providing a computationally efficient, sampling-free uncertainty estimate.

The methodology is structured into four key components:

Coordinate-Wise CP with Bonferroni Correction:
Since bounding box regression involves four coordinates, CP is applied coordinate-wise. To ensure joint box-level coverage guarantees without assuming dependence structures between coordinates, the authors apply a Bonferroni correction, allocating a miscoverage level of $\alpha/4$ to each coordinate.
Scaled vs. Unscaled CP:
- Unscaled CP: Uses a fixed quantile derived from absolute residuals ( $| \hat{f}(x) - y |$ ), resulting in constant-width intervals.
- Scaled CP: Adapts interval width based on input-dependent aleatoric uncertainty estimates ( $\hat{\sigma}(x)$ ). The score function is rescaled: $s(\hat{y}, y) = | \hat{f}(x) - y | / \hat{\sigma}(x)$ .
- Calibration Variants: The study evaluates the raw (uncalibrated) loss attenuation estimates against two calibrated variants using relative isotonic regression (applied globally and per-coordinate/per-class) to correct miscalibration in the raw variance estimates.
Class-Agnostic vs. Class-Wise CP:
- Class-Agnostic: Computes a single set of conformity scores across all classes.
- Class-Wise: Computes separate calibration quantiles per ground-truth class (assuming known labels), allowing interval widths to adapt to class-specific difficulty.
Two-Step Pipeline (Joint Classification and Regression):
Following Timans et al. (2025), the authors implement a pipeline where:
- Step 1: CP is applied to the classification head using Regularized Adaptive Prediction Sets (RAPS) to generate a prediction set for the object class.
- Step 2: The conformalized bounding box is conditioned on this predicted class set. The quantile for the bounding box is selected as the worst case over only the classes in the prediction set, rather than all possible classes.

Key Contributions
The paper makes the following specific contributions:

Systematic Comparison: A comprehensive evaluation of unscaled versus scaled CP for multi-class object detection, assessing both coverage validity and interval sharpness (IoU and interval score) across in-domain and cross-domain settings.
Uncertainty Calibration Analysis: An investigation into how calibrating raw loss attenuation estimates (via isotonic regression) affects the performance of scaled CP.
Class-Wise Evaluation: An analysis of class-wise CP for bounding box regression, demonstrating its impact on coverage and sharpness relative to class-agnostic settings.
Joint Pipeline Evaluation: An assessment of a two-step pipeline that jointly conformalizes classification and regression heads, specifically testing the interaction between RAPS-based class prediction sets and conformalized bounding boxes.

Experimental Results
Experiments were conducted on three autonomous driving datasets: KITTI, BDD, and CODA (the latter serving as a cross-domain evaluation under distribution shift).

Scaled vs. Unscaled Performance: Scaled CP consistently outperformed unscaled CP in terms of sharpness without sacrificing coverage guarantees.
- IoU Improvements: Scaled CP achieved up to 19% higher IoU (e.g., +5.4% on KITTI, +6.3% on CODA for specific variants) compared to unscaled CP.
- Interval Scores: Scaled CP reduced interval scores by up to 39%, indicating a significantly better trade-off between width and coverage.
- Distribution Shift: The benefits of scaling were most pronounced in the cross-domain setting (CODA), where the detector operates under distribution shift, suggesting adaptive intervals are crucial when model performance degrades.
Calibration Impact: The raw (uncalibrated) loss attenuation estimates captured the majority of the performance gains over unscaled CP. Further calibration via isotonic regression provided only marginal additional improvements in sharpness, with negligible effects on coverage.
Class-Wise vs. Class-Agnostic: Class-wise stratification reduced interval scores for both scaled and unscaled variants, with the reduction being more significant for scaled CP. This suggests class-conditional quantiles interact favorably with aleatoric uncertainty scaling.
Two-Step Pipeline: In the joint classification-regression pipeline, scaled CP demonstrated superior robustness. While unscaled CP suffered a substantial increase in interval score (worsening sharpness) when conditioned on class prediction sets, scaled CP's interval score actually decreased (improved). This indicates that scaled CP degrades more gracefully when combined with classification uncertainty.

Significance and Claims
The authors claim that scaling Conformal Prediction intervals by aleatoric uncertainty estimates derived from loss attenuation is a simple, computationally efficient, and principled method to improve the sharpness of conformalized object detectors.

Key takeaways emphasized by the authors include:

Actionable Uncertainty: The proposed pipeline yields more actionable uncertainty estimates for real-time, real-world object detection by avoiding unnecessary over-conservatism in low-uncertainty scenarios.
Robustness: The superiority of scaled CP holds across different datasets, uncertainty calibration strategies, and pipeline configurations (class-agnostic, class-wise, and two-step).
Validity: The improvements in sharpness are achieved without compromising the marginal coverage guarantees, which remain valid for both scaled and unscaled variants.
Practicality: The approach requires only a single forward pass and no architectural changes beyond the output head, making it suitable for safety-critical applications where computational efficiency is paramount.

The paper concludes that while CP provides the necessary coverage guarantees, integrating it with input-dependent uncertainty estimates (scaled CP) is essential for producing tight, useful prediction sets in complex, multi-output tasks like object detection.