🤖 AI

Respecting Modality Gap in Post-hoc Out-of-distribution Detection with Pre-trained Vision-Language Models

This paper proposes an online pseudo-supervised framework that learns visual class prototypes from unlabeled test data to bridge the intrinsic modality gap between text and visual representations, thereby achieving state-of-the-art post-hoc out-of-distribution detection without requiring in-distribution training data.

Original authors: Yuanwei Hu, Bo Peng, Yadan Luo, Zhen Fang, Ling Chen, Jie Lu

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuanwei Hu, Bo Peng, Yadan Luo, Zhen Fang, Ling Chen, Jie Lu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read librarian (the AI model) who has spent years studying a massive library of books about specific topics, like "cats," "cars," and "trees." This librarian is excellent at identifying these things. However, in the real world, people might walk in and show the librarian a picture of a "glittery toaster" or a "flying banana." These are things the librarian has never seen before.

The goal of Out-of-Distribution (OOD) detection is to teach the librarian to say, "I don't know what this is," instead of guessing it's a cat or a car and getting it wrong.

The Old Way: The "Textbook" Problem

Recently, scientists discovered a new tool: a Vision-Language Model (like CLIP). Think of this as a librarian who can read both pictures and words. To help the librarian recognize "cats," the old method simply wrote the word "cat" on a card and told the librarian, "This word represents the idea of a cat."

The problem is that words and pictures are different languages.

The word "cat" lives in the world of text.
The picture of a cat lives in the world of images.

The paper argues that simply using the word "cat" as a stand-in for the picture of a cat is like trying to navigate a city using a map of a different city. Even though the map and the city share some names, the streets don't line up perfectly. There is a "Modality Gap"—a structural mismatch between the text description and the actual visual reality. No matter how cleverly you rewrite the prompt (the instructions), you can't fix this gap because the text and the image just don't live in the same "neighborhood" in the AI's brain.

The New Solution: Learning by Doing (The "Online" Approach)

The authors propose a new way to fix this. Instead of relying on the static text cards, they let the librarian learn the visual map directly while they are working.

Here is how their method works, using a simple analogy:

The Setup: The librarian starts with the text cards (the old way) as a rough starting point.
The Stream: As people walk in with new pictures (test data), the librarian looks at them.
The Guess: The librarian makes a "soft guess" based on their current knowledge. "This looks 80% like a cat, 20% like a dog."
The Update:
- If the librarian is very confident it's a cat, they slightly adjust their mental "cat map" to match the actual picture they just saw.
- If the librarian is very confident it's not a cat (it's an unknown object), they adjust their "unknown object map."
- If the librarian is confused, they ignore the picture for now.
The Result: Over time, the librarian stops relying on the imperfect text cards and builds a perfect, visual map of what "cat" and "unknown" actually look like in the real world.

Why This Matters

The paper proves mathematically that the old way (using text as a prototype) has a permanent flaw because text and images are fundamentally different. Their new method fixes this by calibrating the map on the fly using the pictures themselves, without needing any human labels or retraining the whole system.

The Proof

They tested this on huge datasets (like ImageNet, which has thousands of categories). The results were impressive:

The new method was much better at spotting "unknowns" (like the flying banana) than previous methods.
It reduced the number of times the AI confidently guessed the wrong thing (False Positives) significantly.
It worked well even when the pictures were shuffled in different orders or came from different sources.

In short, the paper says: "Don't just read the label; look at the picture and learn from it as you go." This allows the AI to be much more reliable when it encounters things it has never seen before.

Technical Summary: Respecting Modality Gap in Post-hoc Out-of-distribution Detection with Pre-trained Vision-Language Models

Problem Statement

Out-of-distribution (OOD) detection is critical for ensuring the reliability of machine learning models in open-world settings, where test samples may belong to unknown classes not seen during training. While recent advances in pre-trained Vision-Language Models (VLMs), particularly CLIP, have enabled zero-shot OOD detection without access to in-distribution (ID) training data, existing approaches rely on a specific paradigm: treating text embeddings of class names as class prototypes.

This paper challenges the validity of this "text-as-prototype" paradigm. It argues that existing methods implicitly assume textual representations are well-aligned with optimal visual class prototypes. However, recent evidence suggests a persistent "modality gap" where text and vision representations in CLIP-based models remain distinct. The authors posit that off-the-shelf textual prototypes are generally misaligned with the optimal visual prototypes, creating an intrinsic structural gap that cannot be resolved solely through prompt engineering or negative-label mining. This misalignment leads to sub-optimal OOD detection performance.

Methodology

To address the modality gap under post-hoc constraints (where no ID training data is available), the paper proposes an online pseudo-supervised framework that directly learns and updates class prototypes within the visual feature space.

1. Theoretical Characterization of the Modality Gap

The authors first provide a theoretical analysis under an idealized supervised formulation. They derive a closed-form expression for optimal visual class prototypes, proving that these prototypes must lie entirely within the span of the visual feature space. In contrast, text-derived prototypes often contain components orthogonal to this visual subspace.

Theorem 1: Establishes that optimal visual prototypes are linear combinations of visual features.
Theorem 2: Proves a non-vanishing lower bound on the distance between textual prototypes and optimal visual prototypes whenever the text embeddings contain components outside the visual feature subspace. This formalizes the "modality gap" as a structural limitation.

2. Online Pseudo-Supervised Learning Framework

Since ground-truth labels are unavailable in the zero-shot setting, the method utilizes the strong zero-shot transfer capabilities of pre-trained CLIP models to generate soft pseudo-labels.

Pseudo-Supervised Objective: The framework replaces hard ground-truth labels with soft predictions from the pre-trained VLM to construct a pseudo-supervised loss function.
Online Optimization: An online optimization procedure is developed to incrementally update visual prototypes using unlabeled test-time data streams. This respects the post-hoc constraint by processing data as it arrives.
Prototype Update Strategy:
- The ID label space is extended to include $L$ OOD pseudo-classes (placeholders).
- A hard thresholding rule based on existing OOD scores (e.g., from NegLabel) classifies incoming test samples as "positive" (likely ID) or "negative" (likely OOD).
- ID class prototypes are updated only when a sample is classified as ID; OOD pseudo-class prototypes are updated when a sample is classified as OOD.
- Samples with uncertain scores are ignored.
Convergence Guarantees: Theorem 3 provides theoretical guarantees for the convergence of the proposed online optimization procedure, bounding the cumulative regret by $O(\sqrt{N})$ .

3. OOD Scoring

The final OOD score for a test image is computed by comparing the image's visual feature against the updated visual prototypes (both ID and OOD pseudo-classes) using a softmax-based similarity measure, rather than relying on the original text embeddings.

Key Contributions

Theoretical Insight: The paper theoretically demonstrates that the "text-as-prototype" assumption in CLIP-based post-hoc OOD detection is fundamentally flawed due to an intrinsic modality gap. It proves that text embeddings cannot perfectly represent optimal visual prototypes without visual calibration.
Novel Framework: It introduces an online pseudo-supervised framework that learns class prototypes directly in the visual feature space using unlabeled test streams and soft VLM predictions, effectively bridging the modality gap.
Theoretical Guarantees: The authors provide rigorous convergence guarantees for their online optimization algorithm.
State-of-the-Art Performance: The method achieves new state-of-the-art results across various OOD detection benchmarks, including ImageNet-1K, OpenOOD, and CIFAR datasets, outperforming strong baselines like AdaNeg and NegLabel.

Experimental Results

Extensive experiments validate the effectiveness of the proposed method:

ImageNet-1K: The method achieves an AUROC of 97.75% and an FPR95 of 12.79%, surpassing the previous state-of-the-art (AdaNeg) by significant margins (e.g., +1.09% AUROC and -6.13% FPR95).
OpenOOD Benchmark: Consistent improvements are observed on both Near-OOD and Far-OOD datasets, demonstrating robustness across different types of distribution shifts.
CIFAR Benchmarks: The method shows consistent gains on CIFAR-10 and CIFAR-100 setups.
Ablation Studies: The method is shown to be robust to hyperparameter choices, effective across different visual encoder architectures (ResNet-50, ViT-B/32, ViT-L/14), and scalable to larger input resolutions.
Domain Generalization: The approach maintains superior performance when ID data undergoes domain shifts (e.g., ImageNet-A, ImageNet-R, ImageNet-S).

Significance and Claims

The paper claims that its primary significance lies in identifying and addressing a structural limitation in the current paradigm of zero-shot OOD detection. By moving away from the reliance on text embeddings as static prototypes and instead learning dynamic, calibrated visual prototypes from test-time data, the method offers a more theoretically grounded and empirically effective solution.

The authors emphasize that their approach is not merely an engineering tweak but a fundamental shift in how prototypes are constructed for VLM-based OOD detection. They assert that the modality gap is a structural issue independent of negative-label mining strategies, and their framework provides a principled way to mitigate this gap, leading to consistent performance gains across diverse benchmarks without requiring retraining on ID data.