🤖 machine learning

Vendi Novelty Scores for Out-of-Distribution Detection

This paper introduces the Vendi Novelty Score (VNS), a linear-time, non-parametric out-of-distribution detector that leverages diversity metrics to achieve state-of-the-art performance across various benchmarks while remaining effective even with minimal training data.

Original authors: Amey P. Pasarkar, Adji Bousso Dieng

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Amey P. Pasarkar, Adji Bousso Dieng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, highly trained security guard at the entrance of a museum. This guard has spent years studying thousands of paintings from the Renaissance. They can instantly tell you if a new painting is a genuine Rembrandt or a fake. But what happens if someone walks up with a picture of a toaster, a cartoon cat, or a modern abstract sculpture?

The guard might still say, "I'm 99% sure this is a painting!" because they are so confident in their training, even though they are completely wrong. In the world of Artificial Intelligence, this is called an Out-of-Distribution (OOD) problem. The AI is seeing something it has never seen before, but it doesn't know it's confused.

This paper introduces a new tool called Vendi Novelty Score (VNS) to help AI systems realize when they are looking at something totally foreign. Here is how it works, explained through simple analogies.

The Old Way: Guessing Confidence

Most current methods try to detect these "strangers" by asking the AI, "How confident are you?"

The Flaw: If the AI is trained on cats and dogs, and you show it a toaster, it might still feel very confident because the toaster shares some visual features (like edges or colors) with a cat's ear. It's like a guard who is so used to seeing red coats that they think a red fire hydrant is a person.

The New Way: Measuring "Crowd Diversity"

The authors of this paper decided to stop asking "How confident are you?" and start asking, "How much does this new person change the crowd?"

They use a concept called Vendi Score, which is basically a fancy way of measuring diversity.

The Analogy: The Party Guest List

Imagine the AI has a memory of all the people it has met (the training data).

The In-Distribution (ID) Crowd: These are the people the AI knows well. They are grouped into "cliques" (like a group of all redheads, a group of all tall people, etc.).
The Test Sample: A new person walks in.

How VNS works:

The Local Check (The Clique): The AI looks at the specific clique the new person might belong to (e.g., "Is this person a redhead?"). It asks: "If we add this new person to the redhead group, does the group become more diverse or more repetitive?"
- If the new person is a redhead with a unique hairstyle, the group becomes more diverse.
- If the new person is a redhead who looks exactly like everyone else, the group doesn't change much.
- The Twist: If the new person is actually a toaster (or a cartoon cat), adding them to the "redhead" group creates a weird, jarring mix. The "diversity score" spikes because the toaster doesn't fit the pattern at all.
The Global Check (The Whole Party): The AI also looks at the entire party. "If we add this toaster to the whole room, does the room feel chaotic?"
- VNS calculates how much the new sample disrupts the overall harmony of the data.
The Final Score: The AI combines these two checks. If the new sample makes the local group weird and the global party chaotic, the Vendi Novelty Score goes up. A high score means: "Hey, this doesn't belong here! We should be careful."

Why This is a Big Deal

The paper claims this method is a game-changer for three main reasons:

1. It's Fast and Lightweight
Calculating diversity usually requires heavy math that takes a long time (like checking every single person in a stadium against every other person).

The Paper's Trick: They found a shortcut. Instead of looking at the whole crowd, they only look at the "most important" person in the group (the mathematical equivalent of the "main leader" of the clique).
The Result: It's incredibly fast. The paper shows it runs almost as quickly as the simplest methods, making it practical for real-time use (like in self-driving cars).

2. It Doesn't Need a Full Memory
Usually, to check if something is new, you need to remember every single example you've ever seen. That takes up a lot of computer memory.

The Paper's Claim: VNS works amazingly well even if you only show it 1% of the training data. It's like a security guard who can spot a stranger even if they only remember 1% of the people they've met, because they understand the structure of the crowd, not just the faces.

3. It Works Everywhere
The authors tested this on many different types of AI "brains" (architectures) and many different types of "museums" (datasets like CIFAR-10, CIFAR-100, and ImageNet).

The Result: VNS consistently beat or matched the best existing methods. It was particularly good at spotting "Near-OOD" items—things that look almost like what the AI knows but aren't quite right (like a picture of a dog that looks a bit like a wolf).

Summary

The Vendi Novelty Score is a new way to tell an AI when it's out of its depth. Instead of just asking "Are you sure?", it asks, "Does this new thing fit the pattern of the group?"

By measuring how much a new sample disrupts the diversity of the AI's memory, it can spot strangers (like toasters in a cat museum) with high accuracy, using very little memory, and very quickly. This helps keep AI systems safe by knowing when to say, "I don't know what this is, let a human handle it."

Technical Summary: Vendi Novelty Scores for Out-of-Distribution Detection

Problem Statement

Modern machine learning systems often exhibit unpredictable behavior when encountering out-of-distribution (OOD) inputs, frequently producing high-confidence but incorrect predictions. This poses significant risks in high-stakes domains such as autonomous driving, medical imaging, and scientific discovery. While OOD detection is critical for safety, existing post-hoc methods face limitations:

Logit-based methods (e.g., Maximum Softmax Probability, Energy scores) ignore in-distribution feature statistics, reducing reliability.
Feature-space density methods (e.g., Mahalanobis distance) often rely on restrictive assumptions about the geometry of representations (e.g., Gaussianity, neural collapse) that may not hold across diverse architectures.
Similarity-based methods (e.g., k-Nearest Neighbors) incur significant computational overhead at inference time.
Gradient-based methods often require multiple forward and backward passes, making them impractical for latency-sensitive applications.

There is a need for a computationally efficient, post-hoc OOD detector that does not rely on restrictive distributional assumptions and can operate effectively even with limited access to training data.

Methodology: The Vendi Novelty Score (VNS)

The authors propose the Vendi Novelty Score (VNS), a post-hoc OOD detector grounded in the concept of diversity rather than density estimation. VNS leverages the Vendi Score (VS), a family of similarity-based diversity metrics, to quantify how much a test sample increases the diversity of an in-distribution (ID) feature set.

Core Algorithm

Class-Conditional Novelty:
- For a trained classifier $f_\theta$ with $C$ classes, the method extracts embeddings $h(x)$ from the penultimate layer.
- For each class $c$ , a class-conditional density matrix $\rho_c$ is computed from the training embeddings of that class.
- The novelty of a test sample $x$ with respect to class $c$ is defined as the log-ratio increase in the Vendi Score of order 2 ( $VS_2$ ) when $x$ is added to the class set:
  $\Delta_c(x) = \log VS_2(\rho'_c) - \log VS_2(\rho_c)$
- To ensure efficiency, the authors employ a rank-1 approximation. Instead of maintaining the full eigenspectrum of $\rho_c$ , they approximate the change in $VS_2$ using only the largest eigenvalue $\lambda_c$ and its corresponding eigenvector $u_c$ . This reduces the complexity of computing class-conditional scores to $O(D)$ per class.
Probability-Weighted Aggregation:
- Rather than relying on a single predicted class, VNS aggregates class-conditional novelty scores using the model's predicted class probabilities $p(x)$ .
- The local OOD score is computed as:
  $S_{\text{LOCAL-OOD}}(x) = \sum_{c \in T_K(x)} N_c p_c(x)^\gamma \Delta_c(x)$
  where $T_K(x)$ represents the top- $K$ predicted classes, $N_c$ is the class size (used for scaling), and $\gamma$ controls the sharpness of the aggregation.
Global Diversity Correction:
- To account for global density variations, a global novelty score is computed based on the change in $VS_\infty$ (which depends on the maximum eigenvalue of the global density matrix) when the test sample is added to the entire dataset.
- The final VNS score combines local and global signals:
  $\text{VNS}(x) = S_{\text{LOCAL-OOD}}(x) - g \cdot S_{\text{GLOBAL-OOD}}(x)$
  where $g \in \{0, 1\}$ is a binary coefficient determining whether the global correction is applied.

Complexity

Time Complexity: $O(CD)$, where $C$ is the number of classes and $D$ is the embedding dimension. This is linear in both parameters, making it significantly faster than Mahalanobis-based methods ( $O(CD^2)$ ) and k-NN approaches.
Space Complexity: $O(CD)$, as it only requires storing the top eigenvalue and eigenvector for each class.

Key Contributions

New Paradigm: Introduces a diversity-based perspective for OOD detection, moving away from traditional density modeling or logit analysis.
Vendi Novelty Score (VNS): Proposes a specific algorithm that quantifies novelty via the impact of a test sample on representation diversity, utilizing efficient rank-1 approximations of the Vendi Score.
Hybrid Signal Integration: Combines local (class-conditional) and global novelty signals, weighted by predictive probabilities, to improve robustness.
Data Efficiency: Demonstrates that VNS retains state-of-the-art performance even when computed using only 1% of the training data, enabling deployment in memory-constrained settings.
Theoretical and Empirical Validation: Provides theoretical justification for the rank-1 approximation and extensive empirical validation across multiple datasets and architectures.

Experimental Results

The authors evaluated VNS on standard OOD benchmarks: CIFAR-10, CIFAR-100, and ImageNet-1K, using architectures including ResNet, ViT, and Swin-T.

Performance: VNS achieved state-of-the-art (SOTA) performance on CIFAR-10 and competitive results on CIFAR-100. On the ImageNet-1K benchmark, VNS reduced the False Positive Rate at 95% True Positive Rate (FPR@95) by 13% relative to the second-best performing algorithm on average.
Robustness: Unlike some activation-shaping methods that perform well on ResNet but degrade on Transformers, VNS demonstrated consistent performance across different architectures (ResNet, ViT, Swin-T).
Efficiency: VNS is orders of magnitude faster than the second-most accurate method (RMDS++) and comparable in latency to logit-only methods (e.g., GEN, NCI).
Data Scarcity: When trained on only 1% of the ImageNet training data, VNS achieved an average AUROC of 88.96%, nearly identical to its performance with full data access (88.59% AUROC).

Significance and Claims

The paper claims that VNS offers a principled, efficient, and robust solution for OOD detection. Its significance lies in:

Avoiding Restrictive Assumptions: By relying on diversity metrics rather than Gaussian density assumptions, VNS is less sensitive to violations of geometric assumptions common in deep learning representations.
Practical Deployment: The linear-time complexity and ability to function with minimal training data make VNS suitable for real-world scenarios where training data access is limited or latency is critical.
Safety: The method provides a mechanism for ML systems to identify inputs beyond their competence, triggering safer responses such as abstention or human escalation.

The authors note limitations, including the need for hyperparameter tuning (which they address via an automated validation set) and the reliance on the cosine kernel. They emphasize that while VNS improves safety, it is not a complete solution and should be part of a broader safety protocol involving monitoring and human oversight.