🤖 machine learning

Scaling Pretrained Representations Enables Label-Free Out-of-Distribution Detection Without Fine-Tuning

This paper demonstrates that scaling pretrained representations enables accurate, label-free out-of-distribution detection without fine-tuning by revealing that frozen models inherently encode sufficient geometric structure, causing performance gaps between local and global detectors to vanish as representation quality improves.

Original authors: Brett Barkley, Preston Culbertson, David Fridovich-Keil

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Brett Barkley, Preston Culbertson, David Fridovich-Keil

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Overconfident Expert"

Imagine you have a brilliant expert who has studied thousands of pictures of cats and dogs. If you show them a picture of a cat, they say, "That's a cat!" with 100% confidence. But if you show them a picture of a toaster, they might still say, "That's a very strange cat!" with just as much confidence.

In the world of AI, this is called the Out-of-Distribution (OOD) problem. Deep learning models are great at recognizing what they've seen before, but they are terrible at realizing when something is completely new or weird. They don't have a "stop and think" button; they just guess, often incorrectly.

The Old Way: Hiring a New Specialist

Previously, to fix this, researchers thought you needed to do two things:

Give the model labels: Tell it exactly what every picture is (e.g., "This is a cat," "This is a dog").
Fine-tune it: Train the model specifically to spot the weird stuff, essentially hiring a new specialist for every new job.

This is slow, expensive, and requires a lot of data that often doesn't exist (like labeled examples of "weird" inputs).

The New Discovery: The "Frozen" Model is Already Smart

This paper asks a simple question: Do we actually need to retrain the model?

The authors tested modern, pre-trained AI models (like DINO for images and Qwen for text) that are "frozen." This means they are already trained and we aren't allowed to change their internal brain. They just use the "thoughts" (representations) the model generates when it looks at data.

They found that these frozen models already have a hidden "geometric map" inside them. Even without being taught what is "weird," the way the model organizes its thoughts naturally separates normal things from weird things.

The Two Detectors: The "Global Map" vs. The "Local Scout"

To test this, the authors used two different tools to look at the model's thoughts:

The Global Map (Mahalanobis): Imagine looking at a crowd from a drone. You draw a big circle around the "normal" people. If someone is far outside that circle, they are weird. This method looks at the big picture and the overall shape of the data.
The Local Scout (ReSCOPED): Imagine a scout walking through the crowd, checking the texture of the ground and the specific details of every person's face. This method uses a "diffusion" process (think of it like slowly cleaning a dirty window) to see if the details look "typical" or "atypical."

The Main Finding: Bigger is Better, and Simpler is Enough

The authors tested these tools on 59 different combinations of tasks (vision and language) and model sizes. Here is what they found:

Small Models are Picky: When the AI model is small or not very smart, the "Global Map" and the "Local Scout" disagree. Sometimes the Map is right; sometimes the Scout is right. You have to guess which tool to use for the specific job.
Big Models are Consistent: As the models get bigger and smarter (scaling up), both tools get incredibly good at spotting the weird stuff.
The Gap Disappears: In the biggest, most advanced models, it doesn't matter which tool you use. Both the "Global Map" and the "Local Scout" perform almost perfectly. The "weirdness" is so clearly visible in the model's thoughts that even the simplest tool can find it.

The "Sweet Spot" Analogy

The paper also discovered a "sweet spot" in how the "Local Scout" (ReSCOPED) works.
Imagine you are trying to hear a specific sound in a noisy room.

If you turn the volume up too high (too much noise), you hear everything and can't tell what's important.
If you turn it down too low (too little noise), you miss the details.
The authors found that there is a perfect middle volume where the "weird" signals become clear. Surprisingly, this "perfect volume" is the same whether you are looking at pictures or reading text. It's a universal setting for modern AI.

Real-World Use: The "Gatekeeper"

Because these models are already smart enough to spot the weird stuff without retraining, the authors showed how to use them as a gatekeeper.

Imagine a chatbot. Before the chatbot starts writing a long answer, this "gatekeeper" checks the user's question.

If the question looks normal, the gatekeeper says, "Go ahead."
If the question looks weird or dangerous, the gatekeeper says, "Stop! Don't answer this."

The cool part is that this gatekeeper is lightning fast. It adds only about 5 milliseconds (0.005 seconds) of delay. It doesn't need to retrain the chatbot or ask for labels; it just uses the chatbot's own "thoughts" to make the decision.

Summary

Old belief: You need to retrain models and use complex tools to catch weird inputs.
New finding: Modern, big, frozen models already have the "weirdness" built into their structure.
Result: You can detect weird inputs using simple tools without any extra training. As models get bigger, the choice of tool matters less because the signal becomes so clear.
Benefit: This allows for fast, efficient safety checks on AI models right out of the box.

Technical Summary: Scaling Pretrained Representations Enables Label-Free Out-of-Distribution Detection Without Fine-Tuning

Problem Statement

Deep learning models frequently fail to signal when inputs fall outside their training data manifold, leading to unreliable predictions under distribution shift. While unsupervised Out-of-Distribution (OOD) detection is valuable for its task-agnostic nature, traditional methods face significant hurdles: likelihood-based estimators often assign higher probabilities to OOD samples than in-distribution (ID) data, and reconstruction-based methods are sensitive to bottleneck tuning.

A prevailing assumption in the field is that effective OOD detection requires either class-conditional modeling (relying on labeled ID data) or supervised fine-tuning of the representation to expose OOD-relevant structure. Furthermore, prior work suggests that detector rankings are inconsistent across datasets, with no single method universally outperforming others. This paper challenges the necessity of labels and fine-tuning, asking whether modern frozen pretrained representations inherently encode sufficient geometric structure for accurate, label-free OOD detection.

Methodology

The authors evaluate this hypothesis across 59 backbone-task pairings spanning vision (DINO, DINOv3) and language (BERT, Qwen3) modalities. They compare two complementary, label-free detectors operating on frozen latent representations:

Global Mahalanobis Estimator: A label-free adaptation of the classic Mahalanobis distance. Instead of fitting class-conditional Gaussians (which requires labels), this method fits a single global Gaussian model to the unlabeled in-distribution latent representations. It calculates the distance of a test point from the global mean, normalized by the global covariance matrix.
ReSCOPED (Local Score-Curvature Estimator): A lightweight, diffusion-based typicality estimator operating at the local level. Building on SCOPED, ReSCOPED trains a small diffusion observer (approx. 8M parameters) on frozen representations. It evaluates local geometry using a score-curvature ratio ( $T(z) = \|s_\theta(z, \sigma)\|^2 / \kappa(z)$ ), where $s_\theta$ is the diffusion score and $\kappa$ is the trace of the score Jacobian (curvature). This acts as a multi-scale filter to expose OOD signals that may not be globally accessible.

Both methods are applied directly to frozen backbones without fine-tuning, using only unlabeled in-distribution data for fitting.

Key Results

1. High Performance Without Fine-Tuning

Modern frozen representations enable strong label-free OOD detection. On vision benchmarks (e.g., CIFAR-10 vs. CIFAR-100), ReSCOPED using a frozen DINOv3-L backbone achieved 94.3% AUROC, closely matching performance from models that required full-parameter supervised fine-tuning. Similarly, in language tasks, scaled models (Qwen3) achieved high AUROC on CLINC150 and SST benchmarks without labels.

2. The "Scaling Regime" Effect

The paper identifies a regime-dependent pattern regarding detector choice:

Weak/Small Representations: In smaller or less structured backbones (e.g., BERT, DINOv3-S), detector performance is dataset-dependent. For instance, ReSCOPED outperformed Mahalanobis on CLINC150, while Mahalanobis outperformed ReSCOPED on SST. No single method was universally best.
Strong/Scaled Representations: As backbone scale increases (e.g., DINOv3-L, Qwen3-4B), the performance gap between the simple global Mahalanobis detector and the complex local ReSCOPED detector becomes negligible. Both methods converge to high performance, suggesting that scaling makes OOD-relevant geometry accessible to both global and local probes.

3. Empirical Regularity in Denoising Scales

Across 59 configurations in both vision and language, the authors observed a consistent "performance plateau" for ReSCOPED at a specific intermediate denoising noise level ( $t \approx 5$ ). This suggests that OOD-relevant structure in pretrained Transformer representations is exposed at a stable, shared scale, regardless of the specific modality or training objective.

4. Practical Efficiency

The proposed approach enables efficient deployment. ReSCOPED adds minimal latency (approx. 5.4 ms for Qwen3-4B on an NVIDIA 4090) because it requires only one forward pass through the frozen encoder, one through the lightweight observer, and a single Jacobian-vector product. This allows for real-time "gatekeeping" during the prefill stage of LLM inference, rejecting out-of-distribution prompts before autoregressive decoding begins.

Key Contributions

Label-Free Viability: Demonstrated that modern pretrained vision and language representations enable strong label-free OOD detection without fine-tuning or class labels, utilizing both local (ReSCOPED) and global (Mahalanobis) geometric views.
Scale-Dependent Detector Choice: Showed that backbone family and scale dictate detector efficacy. In weaker backbones, detector choice is critical and dataset-dependent; in scaled backbones (DINOv3, Qwen3), the gap between simple global and expressive local methods vanishes.
Shared Granularity Regime: Identified a consistent denoising-scale plateau across modalities, indicating that OOD-relevant structure becomes accessible at a shared scale in Transformer representations.
Deployment Pathway: Demonstrated a practical, low-latency deployment pathway for OOD detection on frozen models, including real-time prefill-stage gatekeeping for LLMs.

Significance and Claims

The paper argues that the geometry exposed by frozen pretrained backbones is the primary driver of OOD detection performance, rather than the specific choice of detector or the need for supervised adaptation. As foundation models scale, they naturally expose OOD-relevant deviations in a form accessible to both simple global statistics and complex local curvature measures.

Consequently, the authors claim that for scaled foundation models, the importance of selecting a specific detector diminishes. The focus shifts from engineering complex detectors to understanding when frozen representations already expose the necessary geometry. This finding reduces the computational and data burden of OOD detection, enabling efficient deployment directly on frozen models without the need for task-specific fine-tuning or labeled anomaly data.

Limitations: The authors note that while the study covers a broad set of backbones and datasets, it does not exhaustively cover all foundation models or deployment regimes. The comparison is limited to two complementary label-free methods, and future work is needed to verify if this convergence holds for other detector families or harder distribution shifts where current representations do not yet saturate performance.