Technical Summary: WALDO – Wasserstein-Aligned Localisation for VLM-Based Distributional OOD Detection

1. Problem Statement

The detection of rare pathologies in medical imaging is a critical clinical challenge. While supervised object detection models perform well on known classes, they fundamentally fail to generalize to pathologies absent from training data. Vision-Language Models (VLMs) offer zero-shot reasoning capabilities, yet current state-of-the-art VLMs struggle with anomaly localisation in medical images. On the NOVA benchmark for brain MRI, top VLMs achieve only 24.5–37.7% mAP@30.

The authors identify three structural limitations in current VLM-based localisation:

1. Lack of Normal Context: VLMs attempt to identify anomalies without a reference for "normal" anatomy, analogous to asking a radiologist to diagnose without seeing healthy cases.
2. Reference Mismatch: When references are provided, random sampling often leads to anatomically mismatched views, causing inconsistent comparisons.
3. High Variance: Stochastic sampling in VLMs leads to high prediction variance (standard deviations >5% mAP), and existing self-consistency methods for language tasks do not extend effectively to spatial localisation.

2. Methodology: WALDO

The authors propose WALDO (Wasserstein-Aligned Localisation for Distributional OOD), a training-free framework that reformulates zero-shot localisation as a comparative inference problem. The method proceeds in three stages:

A. Distributional Similarity via Entropy-Weighted Sliced Wasserstein

Instead of using global CLS tokens, WALDO extracts patch-level embeddings using DINOv2 (ViT-B/16) to preserve spatial structure essential for localisation.

• Metric: The method computes the Sliced Wasserstein (SW) distance between the query image's patch distribution and reference healthy images.
• Efficiency: To avoid the $O(T^3)$ complexity of standard Wasserstein distance, it uses $M=100$ random projections, reducing complexity to $O(TM \log T)$.
• Entropy Weighting: Not all patches are equally informative. The method weights patches by local entropy ($H(\phi)$), assigning higher importance to textured regions (likely pathology) and lower weight to homogeneous backgrounds.
  $$SW^{(w)}_2(P, Q) = \left( \sum_{i=1}^T w_i \cdot (p^{(i)} - q^{(i)})^2 \right)^{1/2}$$

B. Goldilocks Zone Reference Selection

A key theoretical insight is the non-monotonic relationship between reference similarity and localisation accuracy (the "Goldilocks effect").

• High Similarity: Anatomically matched but subtle differences may fall below the VLM's discrimination threshold, leading to noisy predictions (high variance).
• Low Similarity: Obvious differences lead to spurious detections at normal anatomical variations (high bias).
• Optimal Zone: References with moderate similarity (30th–70th percentile) minimize the bias-variance trade-off.
• Diversity: From this "Goldilocks zone," references are selected using a Determinantal Point Process (DPP)-like repulsive sampling to ensure diverse anatomical views.

C. Differential Prompting and Self-Consistency Aggregation

• Prompting: The VLM is prompted to perform differential reasoning: "Compare the patient scan to the healthy reference... Identify regions that appear abnormal." This leverages comparative capabilities rather than absolute anomaly detection.
• Aggregation: Predictions from $K$ references are aggregated via weighted Non-Maximum Suppression (NMS). The confidence of each prediction is weighted by the similarity distance ($\tilde{c} = c \cdot \exp(-\lambda \cdot SW_2)$), down-weighting predictions from less similar references. This reduces prediction variance by approximately 40%.

3. Key Contributions

1. Optimal Transport Formulation: A novel reference selection mechanism using entropy-weighted Sliced Wasserstein distance on DINOv2 patch distributions to preserve spatial structure with $O(n \log n)$ efficiency.
2. Theoretical Characterisation of the Goldilocks Effect: The paper theoretically and empirically demonstrates that moderate-similarity references optimise the bias-variance trade-off in comparative visual reasoning, challenging the intuition that "most similar" is always best.
3. Spatial Self-Consistency: An extension of self-consistency to spatial localisation via confidence-weighted NMS, significantly reducing prediction variance.
4. Training-Free Paradigm: A framework that operates without fine-tuning, enabling zero-shot evaluation on previously unseen pathologies.

4. Experimental Results

Experiments were conducted on the NOVA brain MRI benchmark (907 images, 281 rare pathologies) and a subset of VinDr-CXR chest X-rays.

• NOVA Performance: Using Qwen2.5-VL-72B, WALDO achieved 43.5 ± 1.6% mAP@30, a 19% relative improvement over the zero-shot baseline (36.4%). This improvement is statistically significant ($p < 0.01$).
• Cross-Model Generalisation:
  • GPT-4o: Improved from 19.0% to 32.0% mAP@30 (+68% relative).
  • Qwen3-VL-32B: Improved from 20.4% to 32.0% mAP@30 (+57% relative).
  • Gemini-2.0-Flash: Improved from 18.1% to 38.0% mAP@30 (+110% relative).
  • Note: The largest model, Qwen3-VL-235B (MoE), showed a slight performance drop (-12%), which the authors attribute to potential confusion in its expert routing mechanism when processing reference contexts.
• VinDr-CXR: WALDO showed consistent gains across all models (e.g., Qwen3-32B improved from 12.8% to 34.1% mAP@30), though absolute performance was lower (24–35%) than on MRI due to diffuse pathologies and anatomical overlap in X-rays.
• Ablation Studies:
  • Replacing cosine similarity with Sliced Wasserstein provided the largest gain (+6% mAP).
  • Entropy weighting, Goldilocks sampling, and self-consistency each contributed incremental improvements.
  • Differential prompting outperformed direct anomaly detection prompts by +7% mAP.

5. Significance and Claims

The paper claims that WALDO represents a new paradigm for Out-of-Distribution (OOD) detection in medical imaging. Its primary significance lies in demonstrating that providing healthy reference images enables VLMs to perform differential diagnosis via in-context anatomical priors, rather than relying on learned disease representations.

• Clinical Utility: The authors position WALDO as a tool for triage and attention guidance rather than primary diagnosis, noting that it still struggles with very small lesions (<5% image area) and diffuse pathologies.
• Efficiency vs. Accuracy: While the method adds computational overhead (approx. 3x inference time due to multi-image prompting), it achieves substantial accuracy gains without any model training or parameter updates.
• Modality Specificity: The authors acknowledge that performance varies by modality; the method excels on brain MRI (focal intensity changes) but faces challenges on Chest X-rays (high inter-patient variability and overlapping structures).

The work suggests that future directions include learned end-to-end reference selection and uncertainty calibration for clinical deployment, but currently establishes a strong baseline for training-free, zero-shot anomaly localisation.