Technical Summary: GeoFlowVLM

Problem Statement

Standard dual-encoder Vision-Language Models (VLMs), such as CLIP and SigLIP, map images and text to deterministic points on a shared unit hypersphere via $\ell_2$-normalization. While effective for retrieval and zero-shot transfer, this deterministic representation fails to quantify prediction uncertainty. Specifically, it exposes neither aleatoric uncertainty (intrinsic cross-modal ambiguity where one input maps to many valid outputs) nor epistemic uncertainty (lack of support for queries falling outside the training distribution).

Existing post-hoc uncertainty quantification (UQ) methods suffer from two primary limitations:

1. Geometry Agnosticism: Many methods operate in ambient Euclidean space, ignoring the $\ell_2$-normalization constraint that defines the VLM embedding space.
2. Incomplete Uncertainty Modeling: Most methods capture only one type of uncertainty (either aleatoric or epistemic) or fail to respect the hyperspherical geometry for both modalities simultaneously. For instance, ProbVLM captures both but uses a heuristic approach that misses the hyperspherical structure and measures adapter uncertainty rather than backbone density.

The central challenge is to quantify both uncertainty types from frozen VLM embeddings without retraining the underlying encoders, while strictly respecting the product hypersphere geometry ($S^{d-1} \times S^{d-1}$).

Methodology: GeoFlowVLM

The authors propose GeoFlowVLM, a post-hoc probabilistic adapter that learns the joint distribution of paired $\ell_2$-normalized embeddings on the product hypersphere.

Core Architecture and Training

• Riemannian Flow Matching: The model utilizes Riemannian flow matching to learn a joint density $p_\phi(e_I, e_T)$ on the manifold $M = S^{d-1} \times S^{d-1}$. It regresses onto a closed-form geodesic-tangent target, avoiding costly heat-kernel approximations.
• Single Masked Velocity Field: A key design choice is the use of a single neural network with a block-structured binary mask $m \in \{0, 1\}^{2d}$. This single velocity field $v_\phi(z_t, m)$ handles three distinct modes via different masks:
  1. Joint Flow ($m_{joint}$): Generates both image and text embeddings.
  2. Conditional Flow ($m_{I \to T}$): Generates text given a fixed image.
  3. Conditional Flow ($m_{T \to I}$): Generates image given a fixed text.
• Training Objective: The model is trained using a masked conditional flow matching (CFM) objective. The mask restricts supervision to the generated dimensions while holding conditioned dimensions fixed at their observed values ($t=0$). The training employs entropic optimal transport to pair source and target samples, straightening trajectories on the manifold.
• Consistency: Theorem 1 proves that in the population limit, this single masked network recovers valid Riemannian conditional flow velocity fields for both cross-modal directions and the joint distribution without requiring auxiliary consistency losses.

Uncertainty Readouts

From the single trained joint model, two uncertainty metrics are derived:

1. Aleatoric Uncertainty (Retrieval Entropy):

   • Derived from the conditional posterior $p_\phi(e_T | e_I)$ (or vice versa).
   • Samples are drawn from the conditional flow and aggregated over a retrieval gallery to form a discrete posterior distribution $\pi$.
   • The Shannon entropy $H(\pi)$ quantifies retrieval ambiguity.
   • Theoretical Justification: Using Fano's inequality, the entropy provides a rigorous lower bound on the MAP Bayes risk, certifying that high entropy corresponds to high retrieval difficulty.

2. Epistemic Uncertainty (Marginal Typicality):

   • Derived via an exact chain-rule decomposition of the joint Negative Log-Likelihood (NLL):
     $$-\log p_\phi(e_I, e_T) = \underbrace{[-\log p_\phi(e_I) - \log p_\phi(e_T)]}_{\text{Epistemic Typicality}} - \underbrace{\text{PMI}_\phi(e_I, e_T)}_{\text{Cross-modal Matching}}$$
   • The authors argue that the Pointwise Mutual Information (PMI) term is structurally discriminative (measuring compatibility) rather than epistemic.
   • The Epistemic Score is defined as the sum of marginal typicalities: $u_{ep} = -\log p_\phi(e_I) - \log p_\phi(e_T)$.
   • These marginals are computed from the joint model using Bayes' identity and reverse-time integration of the flow ODE, requiring no additional parameters.

Key Contributions

1. GeoFlowVLM Adapter: A post-hoc probabilistic adapter trained via Riemannian flow matching on the product hypersphere using a single masked velocity field. It yields joint, marginal, and conditional densities without encoder retraining. An ablation study confirms that respecting hyperspherical geometry is structurally necessary for calibrated uncertainty.
2. Conditional Retrieval Entropy: A new aleatoric estimator that quantifies cross-modal ambiguity. It tracks retrieval difficulty with near-ideal monotonic calibration across benchmarks and provides a decision-theoretic lower bound on Bayes risk.
3. Chain-Rule Decomposition: A theoretical derivation showing that the joint NLL decomposes into marginal typicality (the correct epistemic signal) and cross-modal PMI (a discriminative signal). This explains the empirical success of per-modality density methods like REPVLM and frames them as a coherent extension of the joint model.

Experimental Results

The method was evaluated on a frozen OpenCLIP ViT-H/14 backbone, trained on Conceptual Captions (CC-1M) and tested zero-shot on seven benchmarks.

• Aleatoric Uncertainty (Retrieval):

  • Evaluated on MS-COCO, Flickr30k, and CUB-200 (both Image-to-Text and Text-to-Image).
  • GeoFlowVLM achieved near-perfect monotonic calibration (Spearman $\rho \approx -1.0$) between predicted entropy and Recall@1, significantly outperforming baselines (ProbVLM, AsymVLM, GroVE).
  • ProbVLM showed positive correlation (uncertainty rising on correct queries), and other baselines showed unstable or non-monotonic trends.

• Epistemic Uncertainty (Zero-Shot Classification):

  • Evaluated on ImageNet-1K, Food-101, CIFAR-100, and ObjectNet.
  • The marginal typicality sum ($u_{ep}$) yielded consistently increasing selective accuracy as coverage decreased (rejecting high-uncertainty samples).
  • Component analysis confirmed that PMI is a poor standalone epistemic score (often rejecting correct predictions), while marginal typicality provides the reliable signal.
  • GeoFlowVLM matched or outperformed REPVLM, with a notable advantage on ObjectNet (viewpoint shift), attributed to the joint model's ability to capture cross-modal structure that independent per-modality flows miss.

• Ablations:

  • Geometry: A Euclidean baseline (identical architecture but operating in $\mathbb{R}^d \times \mathbb{R}^d$) failed catastrophically on both aleatoric and epistemic metrics, confirming the necessity of Riemannian geometry.
  • Efficiency: Reliable estimates were achieved with as few as 10 ODE integration steps and a single Hutchinson probe for divergence estimation.
  • Backbone: Results transferred to the SigLIP backbone, though density-based methods showed degraded performance on ImageNet-1K due to SigLIP's more isotropic embeddings.

Significance and Claims

The paper claims that GeoFlowVLM addresses a critical gap in post-hoc UQ for frozen VLMs by providing a single, geometry-aware model that exposes both aleatoric and epistemic uncertainty.

• Theoretical Insight: The work provides a principled explanation for why marginal typicality is the correct epistemic signal in this setting and why cross-modal PMI should not be used as an uncertainty score. It demonstrates that joint modeling on the product hypersphere is strictly richer than per-modality modeling.
• Practical Utility: The method offers a calibrated, direction-symmetric estimator for retrieval difficulty and a robust density-based score for out-of-distribution detection in zero-shot classification, all without modifying the pre-trained VLM backbone.
• Limitations: The authors note that the consistency theorem is a population-limit result; finite-sample bounds remain open. The epistemic scores are density-based (typicality) rather than Bayesian (parameter posterior variance). Additionally, the method requires one-time training on large-scale data and multiple ODE solves per query, distinguishing it from training-free approaches.

The authors conclude that joint embedding modeling is a general strategy for richer probabilistic reasoning over vision-language representations, particularly for safety-critical applications like medical retrieval and autonomous systems.