Post-hoc Probabilistic Vision-Language Models

1. Problem Statement

Vision-Language Models (VLMs) like CLIP and SigLIP have achieved remarkable success in zero-shot classification, retrieval, and generative tasks by mapping images and text into a joint latent space using cosine similarity. However, these models operate deterministically, meaning they output a single point estimate for embeddings and similarity scores.

This deterministic nature presents two critical issues for real-world deployment:

1. Lack of Uncertainty Quantification: They cannot distinguish between high-confidence correct predictions and high-confidence incorrect predictions (overconfidence), especially under domain shifts or distributional changes.
2. Inefficiency of Existing Solutions: Current methods to add uncertainty (e.g., training probabilistic models from scratch, adding trainable adapters, or test-time adaptation) are computationally expensive, require retraining, or significantly increase inference costs.

The authors aim to develop a method that provides reliable, calibrated uncertainty estimates for pre-trained VLMs without architectural changes, additional training, or significant computational overhead.

2. Methodology: BayesVLM

The proposed method, BayesVLM, is a post-hoc probabilistic framework that approximates the Bayesian posterior of a pre-trained VLM. It relies on three core components:

A. Laplace Approximation (LA)

Instead of training a new model, BayesVLM fits a Gaussian distribution to the posterior of the model parameters centered at the Maximum A Posteriori (MAP) estimate (the pre-trained weights).

• Target: The method focuses only on the last projection layers ($P$ for images and $Q$ for text) of the VLM, treating the deep encoders ($\phi$ and $\psi$) as deterministic and fixed.
• Assumption: To make the problem tractable, the authors assume independence between the image and text modalities. They formulate two separate probabilistic models where the likelihood of one modality is conditioned on the fixed embeddings of the other. This allows for separate posterior inference for $P$ and $Q$.
• Hessian Approximation: Computing the full Hessian is infeasible. The authors use the Kronecker-factored Approximate Curvature (KFAC) to approximate the Generalized Gauss-Newton (GGN) matrix. This factorizes the Hessian into two smaller matrices ($A$ and $B$), preserving a richer posterior structure than diagonal approximations while remaining computationally efficient.

B. ProbCosine: Analytical Uncertainty Propagation

A unique challenge in VLMs is that the final prediction relies on the cosine similarity between two random vectors (the probabilistic image and text embeddings). The distribution of cosine similarity is not Gaussian.

• Solution: The authors derive an analytical approximation called ProbCosine.
• Mechanism: They calculate the first two moments (mean and variance) of the cosine similarity distribution based on the Gaussian distributions of the image and text embeddings.
  • Mean: Approximated using the expected values of the embeddings and their norms.
  • Variance: Derived analytically to capture the spread of the similarity score.
• Result: This allows the uncertainty from the projection layers to be propagated directly to the final similarity score without expensive Monte Carlo sampling.

C. Active Learning Application

The method is applied to Active Learning (specifically Bayesian Active Learning).

• Acquisition Functions: The derived uncertainties are used in acquisition functions like BALD (Bayesian Active Learning by Disagreement) and EPIG (Expected Predictive Information Gain) to select the most informative samples for fine-tuning.
• Online Updates: The framework supports online Laplace updates, allowing the posterior to be refined incrementally as new labeled data is acquired during the active learning loop.

3. Key Contributions

1. BayesVLM Framework: The first post-hoc, training-free method to provide uncertainty estimates for large-scale VLMs (CLIP, SigLIP) by approximating the posterior of the projection layers.
2. Analytical ProbCosine: A novel derivation of the distribution over cosine similarities, enabling efficient uncertainty propagation without Monte Carlo sampling.
3. Model-Agnostic & Efficient: The method requires no architectural changes, no retraining from scratch, and adds negligible computational overhead (<5% runtime for base models, <1% for huge models).
4. Robustness to Closed-Source Data: The authors demonstrate that the Hessian can be estimated using proxy datasets (e.g., CC12M for models trained on LAION-400M) with minimal performance degradation, making the method applicable to proprietary models.

4. Experimental Results

The authors evaluated BayesVLM on multiple zero-shot classification benchmarks (ImageNet-R, CIFAR, Flowers, etc.) and active learning scenarios.

• Uncertainty Calibration:

  • BayesVLM significantly outperforms baselines (Vanilla CLIP, Temperature Scaling, Test-Time Augmentation) in Expected Calibration Error (ECE).
  • It reduces overconfident predictions, particularly on out-of-distribution (OOD) data like ImageNet-R.
  • It achieves comparable or better accuracy (ACC) and Negative Log Predictive Density (NLPD) compared to baselines.

• Active Learning:

  • In cross-domain active learning settings (e.g., OfficeHome, ImageNet variants), using BayesVLM uncertainties (via EPIG/BALD) to select fine-tuning data leads to substantially lower misclassification rates and better-calibrated models compared to random or entropy-based selection.
  • The method effectively identifies informative samples, reducing the number of labeled examples needed for adaptation.

• Efficiency & Robustness:

  • Inference Cost: BayesVLM adds only ~0.11 GFLOPs per image for CLIP-Base, whereas Test-Time Augmentation (TTA) increases costs by ~80x.
  • Proxy Data: Estimating the Hessian on a proxy dataset (CC12M) for a model trained on LAION-400M resulted in only slight degradation, proving robustness for closed-source models.
  • Corruption: The method effectively captures increased uncertainty (variance) as input data is corrupted (e.g., text typos, image noise), demonstrating sensitivity to distribution shifts.

5. Significance

This work bridges a critical gap in the deployment of large-scale Vision-Language Models. By providing a computationally cheap, post-hoc way to quantify uncertainty, BayesVLM enables:

• Safety-Critical Applications: Reliable detection of when a model is unsure, preventing catastrophic failures in autonomous systems or medical diagnosis.
• Efficient Data Annotation: Drastically reducing the cost of fine-tuning VLMs by selecting only the most informative data points for labeling.
• Trustworthy AI: Moving VLMs from "black box" deterministic predictors to probabilistic models where confidence levels are interpretable and calibrated.

The paper establishes that high-quality uncertainty estimation does not require retraining massive models from scratch, but can be achieved through principled Bayesian approximations on the final layers of existing pre-trained architectures.