Mitigating the Modality Gap: Few-Shot Out-of-Distribution Detection with Multi-modal Prototypes and Image Bias Estimation

Technical Summary: Mitigating the Modality Gap in Few-Shot OOD Detection

Problem Statement
Vision-Language Models (VLMs), particularly CLIP, have become prominent for Out-of-Distribution (OOD) detection. However, existing methods relying solely on text prototypes (e.g., MCM) suffer from the "modality gap"—a phenomenon where image and text embeddings occupy distinct regions in the embedding space. This separation causes OOD images to exhibit high similarity to In-Distribution (ID) text prototypes due to spatial closeness rather than semantic relevance, leading to high false positive rates (FPR). While few-shot tuning methods exist, they primarily focus on learning negative text prompts and do not explicitly address the fundamental misalignment between image and text modalities.

Methodology: SUPREME
The authors propose SUPREME (a few-shot tuning framework for OOD detection with CLIP) to mitigate the modality gap. The approach consists of three core components:

1. Multi-Modal Prototypes (Theoretical & Empirical Foundation):
   The paper first establishes that incorporating ID image prototypes alongside ID text prototypes improves OOD detection without additional training. By averaging image embeddings for each class, the authors define a Multi-Modal Prototypes score ($S_{MMP}$). Theoretical analysis (Theorem 1) and empirical results show that this dual-prototype approach increases the score separation between ID and OOD samples, effectively reducing the negative impact of the modality gap.

2. Biased Prompt Generation (BPG):
   To further enhance generalization during few-shot tuning, SUPREME introduces BPG. Unlike previous methods that condition prompts only on learnable contexts and image embeddings, BPG conditions text prototypes on three components:

   • Learnable context tokens.
   • Mapped image embeddings ($m(I)$).
   • A Gaussian-based estimated image domain bias ($\mathcal{N}(\mu, \Sigma)$).
     This bias is estimated from the distribution of ID image embeddings. By incorporating this bias into the prompt generation, the method prevents overfitting on limited few-shot data and aligns text prototypes closer to the true distribution of ID images, thereby enhancing image-text fusion.

3. Image-Text Consistency (ITC):
   To directly minimize the modality gap, SUPREME employs two learnable mappings: an image-to-text mapping ($f_{img-txt}$) and a text-to-image mapping ($f_{txt-img}$). These are optimized using two loss functions while keeping the CLIP backbone frozen:

   • Inter-modal loss ($\ell_{inter}$): Minimizes the distance between mapped embeddings and their original counterparts (e.g., ensuring $f_{img-txt}(I)$ aligns with text prototypes).
   • Intra-modal loss ($\ell_{intra}$): Reconstructs the original embedding from the mapped embedding to prevent information loss (e.g., $f_{txt-img}(f_{img-txt}(I)) \approx I$).

4. Generalized Multi-modal Prototype Score ($S_{GMP}$):
   Building on the multi-modal findings, the authors propose a new OOD scoring function, $S_{GMP}$. This score leverages both uni-modal and cross-modal similarities. It calculates the average of maximum similarities between:

   • The original image embedding ($I$) and text/image prototypes.
   • The mapped image embedding ($I' = f_{img-txt}(I)$) and text/image prototypes.
     This integration ensures that both the original and aligned representations contribute to the final decision, enhancing robustness.

Key Contributions

• Analysis of Modality Gap: The paper provides theoretical and empirical evidence that the modality gap negatively impacts VLM-based OOD detection and that incorporating image prototypes mitigates this effect without training.
• SUPREME Framework: A novel few-shot tuning method comprising BPG (to improve generalization via domain bias) and ITC (to reduce the modality gap via consistency losses).
• New Scoring Metric: The introduction of $S_{GMP}$, which exploits multi-modal prototypes and cross-modal similarities to improve separation between ID and OOD distributions.
• State-of-the-Art Performance: Extensive experiments demonstrating consistent outperformance over existing zero-shot and few-shot OOD detection methods.

Experimental Results
The authors evaluated SUPREME on ImageNet-1k and ImageNet-100 as ID datasets against four diverse OOD datasets (iNaturalist, SUN, Places, Texture) and the OpenOOD benchmark.

• ImageNet-1k: SUPREME achieved an average FPR95 of 20.70% and AUROC of 95.38%, outperforming strong baselines like NegPrompt (23.01% FPR95) and LoCoOp (28.66% FPR95).
• ImageNet-100: The method achieved an average FPR95 of 8.47% and AUROC of 98.08%, significantly surpassing CoOp and CoCoOp.
• Hard OOD (ImageNet-10/20): SUPREME demonstrated robustness in scenarios with high semantic overlap, achieving the lowest FPR95 (4.10% and 3.60%) in both directions.
• Ablation Studies: Removing BPG or ITC resulted in significant performance degradation (e.g., FPR95 rising to ~28%), confirming the necessity of both modules. The use of $S_{GMP}$ over $S_{MCM}$ also yielded measurable improvements.
• ID Accuracy: SUPREME maintained high ID classification accuracy (71.8% Top-1 on ImageNet-1k), outperforming other OOD-focused methods.

Significance and Claims
The paper claims that SUPREME addresses a critical limitation in current VLM-based OOD detection: the modality gap. By moving beyond text-only prototypes and introducing mechanisms to align image and text spaces (via BPG and ITC), the method significantly reduces false positives. The authors emphasize that their approach improves reliability without requiring access to OOD data during training, relying only on a small subset of ID data (16 shots per class). The work suggests that explicitly modeling the relationship between modalities is essential for robust OOD detection in real-world deployments.