🤖 machine learning

Localized Conformal Prediction for Image Classification with Vision-Language Models

This paper presents a localized conformal prediction framework for image classification with vision-language models, demonstrating that while raw cosine similarity fails to improve upon non-local baselines, a proposed simple non-linear transformation of these similarities significantly reduces prediction set sizes while maintaining marginal coverage guarantees.

Original authors: Clément Fuchs, Tim Bary, Benoît Macq

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Clément Fuchs, Tim Bary, Benoît Macq

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are asking a very smart, well-traveled friend (a Vision-Language Model) to identify what's in a photo. They are usually great at it, but sometimes they are unsure. In the real world, it's risky to just guess "It's a cat" when you might be wrong, especially if you're driving a car or diagnosing a patient. You want them to say, "I'm pretty sure it's a cat, but it could also be a fox," or "I have no idea, don't trust me on this."

This is where Conformal Prediction comes in. Think of it as a safety net. Instead of giving a single answer, the system gives a list of possibilities (a "prediction set") that is guaranteed to include the correct answer most of the time (e.g., 90% of the time).

The Problem: The "One-Size-Fits-All" Safety Net

The standard way this safety net works is like a global rulebook. It looks at thousands of past examples (calibration data) and says, "Okay, to be safe 90% of the time, we need to list 5 possibilities for everyone."

The problem? This rulebook is too blunt.

For an easy picture (like a clear photo of a golden retriever), listing 5 possibilities is silly. You only need 1 or 2.
For a tricky picture (like a blurry dog that looks like a wolf), 5 might not even be enough.

The standard method treats a blurry, confusing image the same as a crystal-clear one. It doesn't adapt to the specific situation.

The Proposed Solution: A "Local" Safety Net

The authors tried a smarter approach called Localized Conformal Prediction (LCP).

Imagine instead of a global rulebook, you have a local guide for every single photo. When a new photo arrives, the guide looks at its past experiences and says, "This photo looks a lot like these 10 photos I saw yesterday. For those specific photos, I only needed to list 2 options to be safe. So, for this new one, I'll only list 2."

This is the "Local" part: it adapts the safety net based on how similar the new photo is to previous ones.

The Twist: The "Naive" Guide Failed

The researchers tried to build this local guide using Vision-Language Models (VLMs). These models turn images into mathematical points in a giant space. The most obvious way to measure similarity is to see how close two points are (using cosine similarity).

They thought: "If two images are close in this math-space, they are similar. Let's use that distance to adjust our safety net."

The Result: It didn't work well. In fact, it often made things worse.

The Analogy: Imagine trying to judge how similar two people are just by looking at their height. Two people might be the same height but have completely different personalities, styles, or backgrounds. The "height" metric (cosine similarity) was too simple to capture the true "vibe" of the images in a way that helped the safety net. The "naive" guide gave bad advice, leading to lists that were either too long (wasteful) or too short (unsafe).

The Fix: The "Sigmoid" Filter

The authors realized they needed a better way to measure similarity. They introduced a non-linear transformation (a mathematical filter called a sigmoid function).

The Analogy: Think of the raw similarity score as a dimmer switch that goes from 0 to 100. The "naive" guide just used the number directly. The new method adds a special lens over the dimmer.
- If the images are very similar, the lens makes them look extremely similar (turning the dial up high).
- If the images are just okay similar, the lens makes them look very different (turning the dial down low).
- It creates a sharper distinction between "close enough to trust" and "too far to trust."

By tuning this lens (using a process called cross-validation), they could tell the local guide exactly how to weigh past examples.

The Outcome

When they tested this new "lens" approach on 9 different types of image datasets (ranging from cars and pets to flowers and satellite photos):

Smaller, Smarter Lists: The new method consistently produced smaller lists of possibilities than the old "one-size-fits-all" method, without losing safety. It was more efficient.
The "Naive" Way Failed: Using the raw similarity without the lens actually made the lists bigger and less efficient in many cases.
Safety Remained: The guarantee that the correct answer is in the list (the 90% coverage) stayed the same. They didn't sacrifice safety for efficiency.

What They Didn't Find

The paper also checked if this method fixed a specific problem called "coverage gaps" (where some groups of images, like rare animals, were less safe than common ones).

The Result: The local method did not significantly fix this specific issue. The safety gaps remained roughly the same as the old method. The main win was simply making the lists shorter and more efficient, not necessarily making the safety more equal across all groups.

Summary

The paper is about teaching a smart AI to be more efficient at guessing.

Old Way: "For everyone, list 5 options." (Safe, but often wasteful).
Failed New Way: "Look at how close the images are in math-space and adjust." (Too simple, made things worse).
Successful New Way: "Look at how close the images are, but use a special mathematical filter to sharpen that view, so we only list options when we are truly confident." (Safe, efficient, and works well).

They released their code so others can use this "special filter" to make their own AI predictions more efficient.

Technical Summary: Localized Conformal Prediction for Image Classification with Vision-Language Models

1. Problem Statement

Conformal Prediction (CP) is a statistical framework widely used for uncertainty quantification in machine learning, offering strong marginal coverage guarantees (i.e., the prediction set contains the true label with a user-defined probability $1-\alpha$ averaged over the entire data distribution). However, a well-known limitation of standard CP is its inability to guarantee conditional coverage for specific subgroups (e.g., specific classes or feature regions). This leads to coverage disparities where certain subpopulations are under- or over-covered.

While Localized Conformal Prediction (LCP) has been proposed to address this by weighting calibration examples based on their similarity to the test sample, its application has been largely restricted to regression tasks and synthetic datasets. The application of LCP to image classification, particularly using Vision-Language Models (VLMs), remains under-explored. A critical open question is whether the intuitive choice of using cosine similarity between visual features in VLM latent spaces is sufficient to improve upon non-local baselines in classification tasks.

2. Methodology

2.1 Framework and Setup

The authors apply the LCP framework to classification tasks using VLMs (specifically CLIP-based models with ViT and CNN backbones). The process involves:

Calibration: A set of calibration samples is used to compute conformal scores.
Scoring: Four standard conformal classification scores are evaluated:
- TopK: Uses the rank of the true class in the prediction vector.
- LAC (Least Ambiguous set-valued Classifier): Uses $1 - \text{probability of true class}$ .
- APS (Adaptive Prediction Sets): Uses the cumulative sum of sorted probabilities up to the true class.
- RAPS (Regularized APS): Adds a penalty term to APS to control set size.
Localization: Instead of using a uniform distribution over calibration scores, LCP constructs a weighted empirical distribution based on a localizer function $H(f_{test}, f_i)$ , which assigns higher weight to calibration samples similar to the test sample.

2.2 The Core Innovation: Non-linear Transformation of Cosine Similarity

The paper investigates the localizer function $H$ . In VLMs, the cosine similarity between image embeddings ( $f_i^T f_j$ ) is an intuitive measure of visual similarity. The authors test two approaches:

Naive Approach: Using the raw cosine similarity directly as the localizer weight ( $H = f_i^T f_j$ ).
Proposed Approach: Applying a normalized sigmoid transformation to the cosine similarity:
$H(f_i, f_j) = \frac{1 + \exp(-m(1-\tau))}{1 + \exp(-m(f_i^T f_j - \tau))}$
where $m > 0$ controls steepness and $\tau \in [0, 1]$ sets the inflection point.

The hyperparameters $m$ and $\tau$ are tuned via 5-fold cross-validation on the calibration set to minimize the average prediction set size. The final parameters are derived from the median of $m$ and mean of $\tau$ across folds.

2.3 Experimental Design

Datasets: Nine diverse image classification datasets (SUN397, Aircraft, EuroSAT, StanfordCars, Food101, Pets, Flower102, DTD, UCF101).
Models: Four VLM backbones (ViT-B/16, ViT-L/14, RN50, RN101).
Metrics:
- Efficiency: Mean Set Size (MSS).
- Validity: Coverage Gap (CovGap) and Minimum Class-Conditional Coverage (MCCC).
Baselines: Standard non-local CP methods using the same scores.

3. Key Results

3.1 Mean Set Size (Efficiency)

The proposed non-linear transformation significantly improves efficiency compared to both non-local baselines and the naive localized approach.

Performance: The proposed method achieved a statistically significant reduction in mean set sizes for 9 out of 9 datasets across 3 out of 4 backbones (ViT-B/16, ViT-L/14, RN50) when compared to the non-local TopK baseline.
Naive Failure: The naive approach (using raw cosine similarity) often degraded performance, increasing set sizes significantly compared to non-local baselines in several cases (e.g., DTD with ViT-B/16, UCF101 with RN50).
Consistency: The improvement held across different conformal scores (LAC, APS, RAPS), though the magnitude varied.

3.2 Coverage Metrics (Validity)

The results regarding conditional validity were more nuanced:

CovGap: The localized approaches (both naive and proposed) produced CovGap values almost identical to non-local baselines across all datasets and backbones. This suggests that localization alone is not sufficient to minimize the coverage gap for image classification with VLMs.
MCCC: Improvements in the worst-case coverage (MCCC) were inconsistent. While the proposed method improved MCCC in some specific instances (e.g., LAC score on Pets with ViT-B/16), it degraded performance in others (e.g., TopK score on the same dataset).

4. Contributions

The paper makes three primary contributions:

LCP for Classification: It benchmarks LCP for image classification using VLMs and demonstrates that the intuitive use of cosine similarity is suboptimal. It proposes a simple, non-linear sigmoid transformation to enhance the effectiveness of the localizer.
Extensive Benchmarking: It provides a comprehensive evaluation across nine diverse datasets and four VLM backbones, showing that the proposed method consistently reduces average prediction set sizes.
Open-Source Implementation: The authors release the first open-source implementation of LCP for classification, designed to be compatible with a wide range of datasets and models.

5. Significance and Claims

The paper claims that while LCP is a promising framework for adapting prediction sets to local distributional properties, its success in classification tasks with VLMs is highly dependent on the design of the localizer function.

Key Finding: The "naive" intuition of using raw cosine similarity in the visual latent space is insufficient and can even be detrimental to efficiency.
Solution: A simple, tunable non-linear transformation of these similarities is necessary to achieve statistically significant reductions in prediction set sizes while maintaining marginal coverage guarantees.
Limitation: The authors modestly note that while efficiency is improved, the method does not currently solve the problem of conditional coverage disparities (CovGap) for this specific setting, as the coverage gap remains similar to non-local baselines.

The work aims to facilitate the deployment of localized conformal prediction in real-world applications by providing a robust, open-source tool and clarifying the necessary preprocessing steps for VLM-based classification.