Technical Summary: Geometry-Aware Uncertainty Coresets for Robust Visual In-Context Learning in Histopathology

Problem Statement

Vision-Language Models (VLMs) offer a promising avenue for computational histopathology by coupling visual perception with open-ended clinical reasoning. However, their clinical deployment faces two critical barriers:

1. Data Scarcity and Privacy: Fine-tuning VLMs with billions of parameters on scarce, expert-annotated, and privacy-sensitive pathology data is computationally prohibitive and often infeasible.
2. Instability of In-Context Learning (ICL): While ICL allows VLMs to perform tasks via demonstrative image-text pairs without parameter updates, it suffers from high sensitivity to the selection of examples and the phrasing of queries. Existing selection strategies, such as $k$-Nearest Neighbour ($k$NN) retrieval, rely on local query similarity but ignore the global distributional structure of the dataset. Furthermore, current methods often fail to account for the joint vision-text embedding geometry of VLMs, leading to unreliable diagnostics, prompt-induced instability, and overconfident hallucinations—issues particularly damaging in histopathology where class imbalance obscures minority morphologies.

Methodology: GAUC

The authors propose GAUC (Geometry-Aware Uncertainty Coresets), a training-free coreset selection method that operates directly in the pre-trained multimodal embedding space of a frozen VLM ($W_\theta$). Instead of learning new parameters, GAUC selects a compact subset of real, clinically traceable demonstration images ($D$) from a full dataset ($F$) by minimizing a composite objective function consisting of three complementary terms:

1. Geometry-Aware Distributional Fidelity (MMD):
   To prevent the coreset from collapsing into dominant morphological clusters, GAUC enforces that the embedding distribution of the selected coreset mirrors that of the full dataset. This is achieved by minimizing the Maximum Mean Discrepancy (MMD) between the two distributions in a reproducing kernel Hilbert space. This ensures global representativeness across all tissue morphologies.

2. Prompt-Robustness Regularisation (EMID):
   To address sensitivity to textual query variations, GAUC utilizes the Effective Mutual Information Difference (EMID). This term quantifies the degradation in the coupling between the model response and the multimodal input when the prompt is paraphrased. By minimizing an upper bound of EMID (derived via Jensen-Shannon divergence over visual, textual, and response embeddings), the method selects demonstrations that yield consistent outputs regardless of prompt phrasing, explicitly leveraging the VLM's joint vision-text alignment.

3. Predictive Variance Regularisation:
   To suppress overconfident yet unstable predictions (hallucinations), the method penalizes the variance of the model's output probability distribution across class labels. Minimizing this term steers the selection toward demonstrations that produce concentrated, low-entropy predictive distributions, discouraging ambiguity or overconfidence on incorrect classes.

The final coreset $D^*$ is obtained by jointly minimizing:
$$D^* = \arg \min_D \left( \text{MMD}^2(F, D) + \alpha \cdot \text{EMID}_{\text{upper}} + \beta \cdot \text{Var}(D) \right)$$
where $\alpha$ and $\beta$ control the trade-offs. The entire optimization is training-free, requiring only forward-pass embeddings and output log-probabilities.

Key Contributions

1. A Geometry-Aware Objective: A Maximum Mean Discrepancy (MMD) based selection mechanism that preserves the global structure of the full dataset, preventing collapse into local clusters.
2. A Prompt-Robustness Regulariser: An EMID-derived term that quantifies response discrepancies under paraphrased queries, enforcing invariance across both visual and textual modalities.
3. A Variance Regularisation Term: A penalty on predictive uncertainty that minimizes output entropy and suppresses overconfident hallucinations.
4. Training-Free Deployment: A framework compatible with any off-the-shelf VLM that improves accuracy, calibration, and robustness without a single gradient update.

Experimental Results

The method was evaluated on two histopathology benchmarks: CRC-100K (8-class tissue subtyping) and MHIST (binary polyp classification), using Qwen and LLaVA VLM architectures.

• Accuracy and Calibration: On CRC-100K with Qwen (3-shot), GAUC achieved the highest accuracy (0.610) and F1 score (0.588), outperforming the strongest learned baseline (MIMIC) by 1.16 percentage points. It also significantly reduced Expected Calibration Error (ECE) compared to mutual-information aligned retrieval baselines.
• Robustness: GAUC demonstrated the lowest prediction variance under prompt paraphrasing (Var-para) and across independent runs (Var-runs), confirming the effectiveness of the EMID regulariser.
• Hallucination Reduction: GAUC achieved a relative reduction of 1.61% in sentence-level hallucination rates (CHAIRs) compared to MIMIC, indicating that the variance penalty effectively discourages high-entropy states prone to fabrication.
• Qualitative Analysis: Visualizations showed that while $k$NN often selected morphologically redundant examples leading to confident misclassifications, GAUC selected diverse demonstrations spanning multiple tissue classes, resulting in correct diagnoses with higher confidence.
• Ablation Studies: Removing any of the three terms (MMD, EMID, or Variance) resulted in statistically significant performance drops, confirming that all components contribute complementary gains.

Significance and Claims

The paper posits that GAUC establishes a new standard for trustworthy, training-free diagnostic support in computational pathology. By integrating geometry-aware selection with information-theoretic prompt invariance, the method addresses the critical limitations of current ICL approaches: sensitivity to example selection, prompt instability, and overconfidence.

The authors claim that this approach is particularly significant because:

• It utilizes real, clinically traceable demonstration images, unlike dataset distillation methods that synthesize artificial samples.
• It requires no parameter updates, making it immediately deployable with existing pre-trained VLMs without the computational cost of fine-tuning.
• It generalizes naturally to other safety-critical domains where reliable in-context learning is essential but data is scarce or privacy-sensitive.

The work suggests that principled coreset selection in the embedding space can overcome the "black box" nature of prompt engineering, providing a robust mechanism for VLMs to perform high-stakes medical tasks without the risks associated with hallucination and calibration errors.