🤖 machine learning

Decoupled Conformal Optimisation: Efficient Prediction Sets via Independent Tuning and Calibration

The paper proposes Decoupled Conformal Optimisation (DCO), a train-tune-calibrate framework that uses independent data splits for structural selection and calibration to achieve standard finite-sample marginal conformal coverage while significantly reducing prediction set sizes compared to traditional coupled PAC-style methods.

Original authors: Fanyi Wu, Lihua Niu, Samuel Kaski, Michele Caprio

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Fanyi Wu, Lihua Niu, Samuel Kaski, Michele Caprio

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Two-Headed" Problem

Imagine you are a chef trying to create the perfect recipe for a soup. You want the soup to be delicious (efficient/small) but also safe to eat (reliable/covered).

In modern machine learning, we use a method called Conformal Prediction to create "prediction sets." Instead of guessing just one answer (e.g., "It will rain"), the model gives a list of possibilities (e.g., "It will rain, or maybe snow, or maybe hail") and guarantees that the true answer is in that list a certain percentage of the time (e.g., 95% of the time).

The problem the paper addresses is how to make these lists as small as possible without breaking the safety guarantee.

The Old Way: The "Chef Tasting Their Own Soup"

Traditionally, when scientists tried to make these prediction lists smaller (more efficient), they used the same batch of test data for two jobs:

Job A (The Search): Trying out different recipes to see which one makes the smallest list.
Job B (The Safety Check): Using that same data to prove the list is safe.

The Flaw: This is like a chef tasting their own soup to decide if it's ready, then tasting it again to write a health inspection report. Because the chef already tweaked the recipe based on the first taste, the second taste isn't a fair, independent test. The safety guarantee gets shaky.

To fix this, previous methods (called "PAC-style" or "CRC") added a huge "safety margin" to the recipe. They made the soup very bland (the prediction lists became very large) just to be absolutely sure the safety report was correct. It was safe, but not very useful.

The New Way: DCO (The "Three-Room Kitchen")

The authors propose a new method called Decoupled Conformal Optimisation (DCO). They suggest splitting the kitchen into three distinct rooms with three different teams:

Room 1: Training (The Practice Kitchen)
- The model learns the basics here.
Room 2: Tuning (The R&D Lab)
- This team tries out different recipes, spices, and cooking times. They search for the most efficient way to make a small list. They pick the "best" recipe based on this data.
- Crucial Step: Once they pick a winner, they throw away the specific measurements they used to pick it. They only keep the recipe itself.
Room 3: Calibration (The Health Inspector)
- This team gets a fresh, untouched batch of ingredients that the R&D team has never seen.
- They take the "winning recipe" from Room 2 and test it on this new batch.
- Because the recipe wasn't tweaked based on this specific batch, the Health Inspector can give a strict, mathematically perfect safety guarantee.

Why This Matters: The "Fresh Batch" Advantage

By separating the "search for efficiency" (Room 2) from the "safety check" (Room 3), the authors get the best of both worlds:

Safety: They still get the strict mathematical guarantee that the prediction list is correct 95% of the time.
Efficiency: Because the safety team didn't have to worry about the search team "cheating" by over-fitting the data, they don't need to add that huge "safety margin." The prediction lists can be much smaller and more precise.

The Results: Smaller Lists, Same Safety

The paper tested this on real-world data (like predicting house prices, diabetes outcomes, and identifying images).

The Result: DCO produced prediction lists that were smaller (more efficient) than the old "safety-first" methods, while still hitting the target accuracy.
Example: On an image classification task, the old method produced a list of 26.5 possible answers on average. DCO reduced this to 25.3, and the "worst-case" lists (the huge ones) got significantly smaller too.
The Catch: If you skip the "Fresh Batch" step (like the "DirectTune" method in the paper), you might get even smaller lists, but you lose the safety guarantee. It's like serving soup without a health inspection—sometimes it's fine, sometimes it's not.

Summary

The paper argues that you don't need to use the same data to "search for the best model" and "certify the model's safety." By using a fresh, independent dataset just for the final safety check, you can find more efficient models without sacrificing reliability. It's the difference between a chef who tastes their own cooking to write a health report (risky) and a chef who hires a separate, independent inspector to check a fresh batch (safe and efficient).

Technical Summary: Decoupled Conformal Optimisation

Problem Statement
Modern conformal prediction (CP) aims to construct prediction sets that are both valid (providing finite-sample marginal coverage) and efficient (small set sizes or narrow intervals). In Bayesian conformal optimisation, practitioners often seek to improve efficiency by searching over structural choices, such as non-conformity scores, prior hyperparameters, or model architectures.

A critical methodological tension arises in how data is allocated for this process. Standard "coupled" approaches (e.g., Conformal Risk Control (CRC) and Bayesian Quadrature (BQ) based methods) use the same held-out calibration split to both search for an efficient structure/threshold and to certify the final risk or coverage. While this coupling is necessary for high-probability risk control guarantees (PAC-style), the authors argue it is unnecessary when the target is standard finite-sample marginal conformal coverage. Reusing calibration data for both search and certification breaks the exchangeability argument required for standard split-conformal guarantees, often leading to overly conservative thresholds to maintain validity.

Methodology: Decoupled Conformal Optimisation (DCO)
The paper proposes Decoupled Conformal Optimisation (DCO), specifically the DCO-Warmstart protocol, which adopts a "train-tune-calibrate" design principle. This approach explicitly separates the data roles:

Training Split ( $D_{train}$ ): Used to fit the Bayesian model and define the posterior predictive distribution.
Tuning Split ( $D_{tune}$ ): Used exclusively for efficiency-oriented structural selection. The algorithm searches over a candidate class of structures $\Phi$ (e.g., score functions, priors, architectures) and thresholds to minimize the expected prediction set size subject to an empirical miscoverage constraint. Crucially, the threshold found here ( $\hat{\lambda}_{tune}$ ) is used only for ranking candidates and is discarded before deployment.
Calibration Split ( $D_{cal}$ ): Used exclusively to compute the final conformal quantile. Once a single structure $\hat{\phi}_{tune}$ is selected from $D_{tune}$ , the final threshold is calculated as the empirical $(1-\alpha)$ quantile of the non-conformity scores computed on the independent $D_{cal}$ .

This separation restores the standard split-conformal exchangeability logic. Because the final structure is fixed independently of $D_{cal}$ , the standard proof for finite-sample marginal coverage applies directly without requiring a confidence parameter ( $\delta$ ) or multiple-testing corrections.

Key Contributions

Design Principle: The formulation of DCO-Warmstart as a decoupled pipeline where structural choices are selected on an independent tuning split, and the final conformal quantile is computed on a fresh calibration split.
Theoretical Guarantee: Proof that DCO-Warmstart inherits the standard finite-sample marginal coverage guarantee for any candidate class $\Phi$ (finite or infinite), provided the structure is fixed independently of the calibration data. No confidence parameter $\delta$ is required.
Theoretical Comparison: A clarification of the distinction between DCO-Warmstart and coupled methods (CRC/BQ). While DCO targets marginal coverage and CRC/BQ target high-probability risk control, the paper shows that under uniform consistency assumptions on the risk estimator, both approaches converge to the same population threshold $\lambda^*$ .
Sample Complexity: An oracle inequality for the tuning stage, establishing the sample size requirements for the tuning split to reliably select an efficient structure from a finite candidate class before the independent calibration step.

Experimental Results
The authors evaluate DCO-Warmstart across regression (Diabetes, California Housing, Concrete) and classification (ImageNet-A, CIFAR-100) benchmarks, comparing it against BQ/CRC-style baselines and a diagnostic "DirectTune" baseline (which optimizes on $D_{tune}$ but lacks recalibration).

Coverage: DCO-Warmstart tracks the nominal coverage level closely (e.g., 0.801 vs. target 0.8 on ImageNet-A), satisfying the marginal coverage guarantee.
Efficiency: DCO-Warmstart consistently produces smaller prediction sets or narrower intervals compared to coupled BQ/CRC baselines, which tend to be more conservative.
- ImageNet-A: Average set size reduced from 26.52 (BQ) to 25.26; 95th-percentile set size reduced from 58.95 to 53.73.
- Diabetes: Average interval width reduced from 2.098 (BQ) to 1.914.
Ablation Studies: Experiments on split allocation (e.g., 33/67 tuning/calibration ratio) suggest that the tuning split should be large enough to stabilize candidate selection, while the remaining data should be allocated to calibration to minimize quantile variability.

Significance and Claims
The paper claims that for the specific goal of finite-sample marginal conformal coverage, coupling optimisation and calibration on the same data is sufficient but not necessary. By decoupling these steps, DCO-Warmstart allows for an explicit efficiency search (optimising structure and thresholds) while retaining the rigorous, distribution-free validity guarantees of standard split conformal prediction.

The authors emphasize that DCO-Warmstart is not a replacement for CRC/BQ methods when the scientific objective is high-probability risk control (guaranteeing that the risk constraint holds with confidence $1-\delta$ over the calibration sample). However, when the target is standard marginal coverage, DCO-Warmstart offers a theoretically sound and empirically more efficient alternative by avoiding the conservatism introduced by coupling the search and certification processes on a single data split.