Near OOD Detection for Vision-Language Prompt Learning with Contrastive Logit Score

1. Problem Statement

The paper addresses the critical challenge of Near Out-of-Distribution (OOD) Detection within the context of Vision-Language Prompt Learning (specifically using models like CLIP).

• Context: Pre-trained Vision-Language Models (VLMs) like CLIP excel at zero-shot classification. To improve performance in few-shot scenarios, Prompt Learning methods (e.g., CoOp, CoCoOp) optimize continuous context vectors in the word-embedding space rather than retraining the entire model.
• The Gap: While these methods improve in-distribution (ID) classification accuracy, they often fail to detect Near OOD samples. Near OOD refers to data that shares the same domain and visual characteristics as the training data but belongs to unseen classes (semantic shift), as opposed to Far OOD which involves obvious domain shifts (e.g., images of dogs vs. digits).
• Limitations of Existing Methods:
  • Standard OOD scoring functions (e.g., MaxLogit, Energy Score, Mahalanobis) are not designed for the specific architecture of prompt learning models, which lack traditional classification heads and rely on cosine similarity between image and text embeddings.
  • Existing prompt-learning-specific OOD methods often require fine-tuning, architectural changes, or retraining, limiting their "plug-and-play" utility.
  • Current methods struggle to distinguish between atypical ID samples (hard-to-classify in-distribution data) and near OOD samples, as both may exhibit low confidence scores.

2. Methodology: Contrastive Logit Score (CLS)

The authors propose Contrastive Logit Score (CLS), a novel, post-hoc, plug-and-play scoring function that requires no model retraining or architectural modification.

Theoretical Foundation

The method is grounded in hypothesis testing. The goal is to distinguish between:

• $H_{ID}$: The sample comes from the in-distribution (modeled by density $p_{ID}$).
• $H_{OOD}$: The sample comes from the near OOD distribution (modeled by density $p_{OOD}$).

Using a log-likelihood ratio test, the score is defined as $\lambda(x) = \log \frac{p_{ID}(x)}{p_{OOD}(x)}$. Since $p_{OOD}$ is unknown, the authors approximate it using a "null class" concept.

The Scoring Function

The CLS score is derived by contrasting the likelihood of the sample belonging to a known class against its likelihood of belonging to a generic "in-distribution but unknown class" (the null class).

The formula is defined as:
$$\text{CLS}(x) = \max_{i \in \{1, \dots, K\}} \langle I, P_i \rangle - \beta \langle I, V \rangle$$

Where:

• $I$: The image feature embedding.
• $P_i$: The prompt embedding for the $i$-th known class (context vectors + class name).
• $V$: The context vectors learned during prompt training, without any specific class names appended. This represents the generic features of the in-distribution domain.
• $\langle \cdot, \cdot \rangle$: Cosine similarity.
• $\beta$: A scaling factor that adjusts the weight of the null term.

Key Insight:

• The term $\max \langle I, P_i \rangle$ (similar to MaxLogit) measures confidence in specific classes.
• The term $\langle I, V \rangle$ measures how well the image aligns with the general in-distribution features.
• Near OOD samples often align well with the general domain features ($\langle I, V \rangle$ is high) but poorly with specific classes.
• Atypical ID samples might have low specific class confidence but also low alignment with general domain features.
• By subtracting $\beta \langle I, V \rangle$, the method penalizes samples that look like the general domain but don't fit specific classes, thereby separating Near OOD from Atypical ID.

Learning the Scaling Factor ($\beta$)

Since the optimal $\beta$ varies by dataset, the authors propose a closed-form estimation using only the few-shot ID training samples:

1. Compute the MaxLogit score and the Context score ($\langle I, V \rangle$) for the training set.
2. Find the $\beta$ that minimizes the correlation (zeros out the off-diagonal covariance) between these two scores.
3. This is achieved via a simple linear regression-like calculation (Lemma 1 in the paper), ensuring the method is computationally efficient and requires no OOD data during the tuning phase.

3. Key Contributions

1. Novel Scoring Function (CLS): Introduced a post-hoc scoring mechanism specifically tailored for vision-language prompt learning that leverages the contrastive nature of CLIP.
2. Plug-and-Play Efficiency: The method requires no retraining, no architectural changes, and no access to OOD data for calibration. It can be applied to any existing prompt learning model (e.g., CoOp, MaPLe, LoCoOp).
3. Theoretical Justification: Framed the problem within a hypothesis testing framework, introducing a "null class" vector to approximate the OOD likelihood.
4. Dataset-Specific Adaptation: Developed a closed-form solution to estimate the optimal scaling factor $\beta$ using only in-distribution training data.
5. Extensive Validation: Validated across 13 diverse datasets and 8 state-of-the-art prompt learning models.

4. Experimental Results

The authors evaluated CLS on 13 datasets (including ImageNet, Caltech101, Flowers102, CIFAR-10/100, etc.) using 8 different prompt learning backbones.

• Near OOD Detection Performance:
  • CLS achieved up to an 11.67% improvement in AUROC compared to the baseline MaxLogit score.
  • On average, CLS improved AUROC across 100 out of 104 evaluations (13 datasets × 8 models).
  • It significantly reduced the False Positive Rate at 95% True Positive Rate (FPR95).
• Comparison with SOTA:
  • CLS outperformed existing OOD methods tailored for VLMs (e.g., NegLabel, NegPrompt, LoCoOp) and general OOD detectors (e.g., Mahalanobis, Energy Score).
  • It achieved the best overall performance in the "CLIP Prompt Learning" category in Table 3.
• Far OOD Detection:
  • While designed for Near OOD, CLS also showed competitive performance in Far OOD detection, ranking second only to specialized methods like GL-MCM, but with a much smaller performance gap than in the Near OOD setting.
• Computational Overhead:
  • The method adds negligible computational cost as it only involves computing cosine similarities and a simple linear estimation for $\beta$.

5. Significance and Impact

• Safety-Critical Applications: The ability to reliably detect Near OOD samples is crucial for deploying VLMs in real-world scenarios (e.g., healthcare, autonomous driving) where models must recognize when an input is "unfamiliar" even if it looks visually similar to known data.
• Efficiency: By avoiding retraining, CLS makes OOD detection accessible for any existing or future prompt learning model without the heavy computational burden of fine-tuning.
• Generalizability: The approach is model-agnostic regarding the specific prompt learning algorithm, making it a versatile tool for the broader community of Vision-Language Model researchers.

In summary, this paper bridges a critical gap in the deployment of efficient Vision-Language Models by providing a simple, effective, and theoretically grounded method to detect subtle distribution shifts (Near OOD) without compromising the model's classification capabilities.