🤖 machine learning

Empirical Bayes Conformal Prediction for Vision and Language Models

This paper introduces an Empirical Bayes Conformal Prediction framework that leverages $r$ -values to transform score variability into an uncertainty-informed nonconformity score, thereby improving ranking stability and reducing the inclusion of high-variance false candidates in vision and language models while maintaining distribution-free coverage guarantees.

Original authors: Jiapeng Zeng, Yogesh Prabhu, Zhanpeng Zeng, Michael A. Newton, Vikas Singh

Published 2026-05-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiapeng Zeng, Yogesh Prabhu, Zhanpeng Zeng, Michael A. Newton, Vikas Singh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge in a talent show. You have a list of contestants (candidates), and you need to pick a "safe list" of the best performers to advance to the next round. You want to be 95% sure that the actual winner is on your list, but you also want the list to be as short as possible so you don't waste time watching bad acts.

This is exactly what Conformal Prediction (CP) does for AI models. It creates a "safe list" of answers that is guaranteed to contain the correct answer most of the time.

However, standard AI models have a problem: they are sometimes unstable. If you ask the same question twice, or look at the same image twice, the model might give slightly different scores. Sometimes, a bad answer gets a high score just by luck (a "lucky guess"), and a good answer gets a low score just by bad luck.

Standard CP only looks at one of these scores. It's like the judge making a decision based on a single, shaky performance. If the model has a "lucky guess" moment, the bad answer gets on the safe list, making the list longer and less useful.

The New Idea: The "r-value" (The Stability Check)

This paper introduces a new method called Empirical Bayes Conformal Prediction using r-values. Think of the r-value as a "Stability Score" or a "Consistency Badge."

Instead of asking, "How high was the score?" the new method asks, "How consistent was the score?"

Here is how it works using a simple analogy:

The Analogy: The "Lucky vs. Reliable" Student

Imagine two students taking a test:

Student A (The Rock): Gets a score of 90 every single time you ask them to take the test. They are stable and reliable.
Student B (The Rollercoaster): Sometimes gets a 95, sometimes a 40, and sometimes a 92. Their average might be high, but they are all over the place.

Standard CP sees Student B's lucky 95 and says, "Wow, 95 is great! They are in the top group!" It puts Student B on the safe list, even though they are actually unreliable.

The r-value method looks at the whole picture. It sees that Student B is a rollercoaster. It says, "Even though you hit 95 once, you are too shaky to be trusted as a top candidate. You might drop to 40 next time." So, it keeps Student B off the safe list (or pushes them down the list) and keeps Student A (the Rock) at the top.

How It Works in Practice

The researchers tested this on two types of AI:

Vision Models (Image Classifiers): Like a robot looking at a picture of a cat.
- The Trick: They asked the robot to look at the same picture 1,000 times with tiny, random changes (like asking a friend to describe the same picture in slightly different words).
- The Result: If the robot kept saying "Cat" every time, it got a high r-value. If it kept switching between "Cat," "Dog," and "Toaster," it got a low r-value. The method successfully filtered out the "Toaster" guesses that only happened by accident.
Language Models (Chatbots): Like a robot answering a trivia question.
- The Trick: They asked the robot the same question but rephrased it in 20 different ways (e.g., "Who is the president?" vs. "Who leads the country?").
- The Result: If the robot gave a great answer to all 20 versions, it was a "Rock." If it gave a great answer to one version and a nonsense answer to another, it was a "Rollercoaster." The r-value method used this to create a shorter, more accurate list of answers.

The Main Takeaways

It doesn't break the rules: The new method still guarantees that the correct answer is on the list 95% of the time (just like the old method).
It makes lists shorter: By filtering out the "lucky guesses" (high scores that are unstable), the list of candidates becomes smaller and more useful.
It's smart about uncertainty: If the model is very stable (no variability), the new method acts just like the old one. But if the model is shaky, the new method uses that shakiness to its advantage to remove bad candidates.
It works for both images and text: Whether the AI is looking at a photo or reading a sentence, checking for consistency makes the final decision better.

In short, this paper teaches AI to stop trusting a single "lucky" score and start trusting consistent performance. It turns the AI's own "inconsistency" into a tool to make better, safer predictions.

Technical Summary: Empirical Bayes Conformal Prediction for Vision and Language Models

Problem Statement

Conformal Prediction (CP) offers a distribution-free framework to generate prediction sets with guaranteed coverage for modern vision and language models. However, standard CP relies on a single realization of a nonconformity score to rank candidates. This approach ignores epistemic uncertainty and score variability arising from model parameters, training randomness, or prompt phrasing.

When multiple candidate labels or responses receive similar scores, small fluctuations can drastically alter their ranking order. Consequently, standard CP may include high-variance false candidates in the prediction set simply because a single noisy realization made them appear strong, while excluding stable candidates that had a slightly lower but more consistent score. Existing alternatives, such as "average-then-calibrate" (CPavg), smooth out this variability by averaging scores before calibration, thereby discarding information about the stability of a candidate's rank. The core problem is that current CP methods fail to utilize score variability as a signal to distinguish between stable evidence and noise-driven fluctuations, leading to less efficient (larger) and less stable conformal sets.

Methodology: The $r$ -value Framework

The authors propose $CPr$-value, an Empirical Bayes Conformal Prediction framework that converts score variability into an uncertainty-informed nonconformity score called the $r$ -value.

Core Concept

Instead of ranking candidates based solely on an observed score $f(x)_i$ , the method treats the ranking as an Empirical Bayes problem. It assumes each candidate has an unobserved latent score $\theta_i$ (the stable signal) and an observed score $f(x)_i$ which is a noisy realization of $\theta_i$ . The $r$ -value measures the probability that a candidate's latent score belongs to the top-ranked group, accounting for both the estimated score and its uncertainty.

Estimators

The framework provides two estimators for the $r$ -value:

Parametric (Normal-Normal Model): Assumes the latent scores follow a normal distribution and the observation noise is Gaussian. This yields a closed-form expression for the $r$ -value. It explicitly models the trade-off between score magnitude and variance, shrinking high-variance scores toward the empirical mean.
Non-parametric (Posterior-Sampling): An assumption-free Monte Carlo estimator. It ranks candidates across multiple posterior samples (e.g., from model ensembles, MC-dropout, or prompt paraphrasing) and calculates the frequency with which a candidate appears in the top- $k$ positions. This frequency serves as the empirical tail probability, which is then converted into an $r$ -value.

Algorithm

Variability Estimation: Generate multiple model outputs (e.g., via Weighted Bayesian Bootstrap adapters for vision models or prompt paraphrasing for VLMs/LLMs) to estimate the distribution of scores for each candidate.
Calibration: Compute $r$ -values for the true labels in a calibration set. Determine the threshold $\hat{r}$ as the $\lceil(n+1)(1-\alpha)\rceil$ -th smallest $r$ -value.
Prediction: For a test input, compute $r$ -values for all candidates. Include candidates with $r(x, u_j) < \hat{r}$ in the prediction set.

Key Contributions

Theoretical Guarantee: Under exchangeability, using the $r$ -value as the nonconformity score preserves the target conformal coverage guarantee ( $1-\alpha$ ).
Efficiency Improvement: Theoretically, the $r$ -value reduces the inclusion probability of high-variance false candidates compared to standard CP and CPavg. This leads to smaller expected prediction set sizes, particularly when variability is informative.
Unified Framework: The method unifies parametric and non-parametric approaches, allowing application to diverse score types (logits, probabilities, discrete evaluator scores) and model architectures (ViTs, CLIP, LLMs).
Ranking Stability: By penalizing candidates with high score variance, the method produces more stable rankings, ensuring that candidates consistently appearing near the top are prioritized over those with sporadic high scores.

Experimental Results

The authors evaluated $CPr$-value across image classification, Vision-Language Models (VLMs), and Large Language Models (LLMs).

Vision Models (Image Classification): Using ViT and ResNet models with Weighted Bayesian Bootstrap (WBB) adapters to approximate posterior variability.
- When variability is small (strong backbone, lightweight adapters), $CPr$-value behaves similarly to standard CP, preserving coverage and set size.
- When variability is informative, $CPr$-value produces smaller and more stable sets, often ranking the true class higher within the set than standard CP.
VLMs and LLMs (CLIP, SigLIP, MobileCLIP, LLaMA): Variability was induced via prompt paraphrasing.
- Coverage: $CPr$-value maintained coverage close to the nominal level (e.g., 95% for $\alpha=0.05$ ).
- Set Size: Significant reductions in set size were observed. For example, on ImageNet, $CPr$-value reduced the average set size by nearly half compared to standard CP.
- Model Strength Trend: The efficiency gains were most pronounced for models with lower accuracy and higher inherent variability (e.g., SigLIP2). As model accuracy increased and variability decreased (e.g., MobileCLIP2), the gains diminished, and the method naturally reverted to CP-like behavior, consistent with theoretical predictions.
- Open-Ended Tasks: In LLM tasks with coarse evaluator scores (e.g., TruthfulQA), the $r$ -value successfully broke ties between responses with identical average scores by leveraging score stability across paraphrases, providing finer ranking resolution.

Significance and Claims

The paper claims that $CPr$-value transforms a perceived weakness of large models—their inconsistency or variability—into a useful signal for conformal efficiency.

Robustness: It provides a method to construct prediction sets that are not only statistically valid but also more efficient by filtering out unstable, noise-driven candidates.
Practicality: The framework is applicable to both vision and language domains, utilizing existing techniques for uncertainty estimation (adapters, ensembles, prompt variations) without requiring retraining of the base models.
Limitations: The authors note that the effectiveness of the method depends on the quality of the variability estimation. If the estimated variability is poor or negligible (as in highly stable models), the benefits are modest. Additionally, the method introduces a computational overhead for generating multiple samples, though this is often feasible via lightweight adapters or parallel inference.

In summary, $CPr$-value offers a principled way to integrate epistemic uncertainty into conformal prediction, yielding smaller, more stable, and more reliable prediction sets for modern AI systems.