💬 NLP

Dynamic Decision Learning: Test-Time Evolution for Abnormality Grounding in Rare Diseases

This paper introduces Dynamic Decision Learning (DDL), a framework that enhances frozen large vision-language models to robustly ground abnormalities in rare diseases by iteratively refining decisions across language and visual spaces, thereby significantly improving localization accuracy and confidence calibration without requiring supervised fine-tuning.

Original authors: Jun Li, Mingxuan Liu, Jiazhen Pan, Che Liu, Wenjia Bai, Cosmin I. Bercea, Julia A. Schnabel

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jun Li, Mingxuan Liu, Jiazhen Pan, Che Liu, Wenjia Bai, Cosmin I. Bercea, Julia A. Schnabel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "One-Glance" Doctor

Imagine you have a very smart, well-read medical student (a Large Vision-Language Model or LVLM) who has studied millions of textbooks. They are great at spotting common things like a broken arm or a standard tumor.

However, when you show them a rare, weird disease they've never seen before, they get shaky. If you ask them the same question in slightly different words, or if you tilt the X-ray image just a tiny bit, their answer changes completely. Sometimes they point to the right spot, but other times they point to nothing at all, or they make up a disease that isn't there (a "hallucination").

The paper argues that relying on a single, quick glance from this AI is dangerous for rare diseases. It's like asking a tourist to navigate a city they've never visited with just one map; if the map is slightly blurry or the street signs are confusing, they get lost.

The Solution: Dynamic Decision Learning (DDL)

The authors propose a new way to use these AI models without retraining them (which is impossible for rare diseases because there isn't enough data). Instead of changing the AI's brain, they change how the AI thinks during the test.

They call this Dynamic Decision Learning (DDL). Think of it as turning a "one-shot" exam into a "team huddle."

1. The Language Coach (DAPE)

First, the system acts like a coach for the AI's instructions.

The Analogy: Imagine you are trying to get a very literal robot to find a specific type of mushroom in a forest. If you say "Find the mushroom," it might miss it. If you say "Look for the brown, round thing on the left," it might find it.
How it works: The system tries out dozens of different ways to ask the question (prompts). It tests them on a small practice set, sees which ones work best, and then uses a "Meta-Coach" (another AI) to combine the best parts of the winning questions into one perfect instruction. It's like refining a recipe until it's perfect before cooking the main meal.

2. The Visual Huddle (V-PUP & RHC)

Once the AI has the perfect question, it doesn't just look at the image once. It looks at it multiple times in different ways.

The Analogy: Imagine a detective looking at a crime scene photo. If they just look at it once, they might miss a clue. But if they rotate the photo, zoom in, flip it, and look at it again, the real clues stay in the same place, while the "ghosts" (illusions or mistakes) disappear.
How it works:
- Perturbation: The system takes the medical image and creates 7 slightly different versions (rotated, flipped, zoomed).
- Consensus: It asks the AI to find the abnormality in all 7 versions.
- The Filter: If the AI points to a spot in all 7 versions, that spot is likely real. If it points to a spot in only one version, that was probably a mistake (a hallucination). The system then uses a mathematical "matchmaker" (Hungarian algorithm) to align all these different views and agree on the final location.

The Result: A "Confidence Score"

The best part of this system is that it doesn't just give you an answer; it gives you a reliability score.

The Analogy: It's like a weather forecast. A standard AI might say, "It will rain tomorrow." This new system says, "It will rain tomorrow, and I am 95% sure because I checked the clouds from every angle and they all agree."
If the AI is unsure (the views disagree), the score is low, and the doctor knows to double-check. If the views all agree, the score is high.

What the Paper Found

The researchers tested this on brain scans containing both common tumors and 281 different types of rare diseases.

Rare Diseases: For the rare diseases, this method improved the AI's ability to pinpoint the exact location of the problem by up to 105%.
Better than Training: Surprisingly, this "thinking harder" method worked better than actually retraining the AI with new data (which is usually the standard fix). This is huge because you can't retrain AI on rare diseases if you don't have enough pictures of them.
Model Size: The bigger the AI model, the better this system worked. The largest models became very good at knowing when they were confident and when they were guessing.

Summary

In short, the paper says: Don't just ask the AI once. Instead, teach it to ask itself better questions, look at the image from every possible angle, and only trust the answer if all those different views agree. This turns a fragile, guessy AI into a stable, reliable assistant for finding rare diseases.

1. Problem Statement

The paper addresses the critical challenge of clinical abnormality grounding (localizing pathological regions in medical images based on natural language instructions) using Large Vision-Language Models (LVLMs).

Data Scarcity & Long-Tail Distribution: Rare diseases often lack sufficient annotated data for supervised fine-tuning (SFT). Existing models struggle with "long-tail" pathologies where data is sparse or non-existent.
Instability of Frozen LVLMs: Even state-of-the-art frozen LVLMs exhibit high sensitivity to minor variations in prompts (linguistic) and visual inputs (perturbations). A single-pass inference often leads to hallucinations (false positives) or unstable localization, failing to distinguish genuine diagnostic signals from noise.
Limitations of Current Adaptation: Traditional domain adaptation (e.g., SFT, LoRA) requires curated labels for every specific disease and deployment environment, which is impractical for rare diseases. Furthermore, standard Test-Time Adaptation (TTA) methods that update model weights are computationally expensive and unstable for large LVLMs.

2. Methodology: Dynamic Decision Learning (DDL)

The authors propose Dynamic Decision Learning (DDL), a parameter-free, test-time framework that enables frozen LVLMs to refine their decisions through an adaptive inference strategy. Instead of updating weights, DDL optimizes the process of inference by exploring semantic and visual spaces.

The framework consists of two primary components operating in parallel:

A. Language Space: Distribution-Aware Prompt Evolution (DAPE)

Goal: To find the optimal instruction prompt ( $p^*$ ) that minimizes grounding error without training the LVLM.
Mechanism:
1. Seeding & Partitioning: An initial pool of prompts is generated. These are evaluated on a small development set ( $D_{dev}$ ) and partitioned into a "Success Tail" (high-performing prompts) and a "Failure Bulk" (low-performing prompts).
2. Contrastive Refinement: A meta-optimizer LLM ( $\Psi$ ) analyzes the semantic gap between the success and failure sets. It iteratively synthesizes new candidate prompts by amplifying successful linguistic patterns and suppressing failure-inducing ones.
3. Convergence: The process continues until the performance distribution of the prompt pool converges, yielding an optimized prompt $p^*$ tailored to the specific task distribution.

B. Visual Space: Visual-Consensus Verification (V-PUP & RHC)

Goal: To quantify the reliability of a detection by testing its stability under visual perturbations.
Mechanism:
1. Visual-Perception Uncertainty Probing (V-PUP): The input image is augmented with $M$ stochastic, spatial-preserving transformations (e.g., slight rotations, scaling, flipping). The LVLM processes these views using the optimized prompt $p^*$ .
2. Referenced Hungarian Consolidation (RHC):
  - Predictions from augmented views are mapped back to the original coordinate space using inverse transformations.
  - A Reference Anchor is established using the prediction from the original (unperturbed) image.
  - A Bipartite Matching problem (solved via the Hungarian algorithm) aligns augmented predictions to the anchor. This filters out uncorrelated noise and hallucinations that do not persist across views.
3. Consensus Reliability Score ( $\sigma$ ): A score is computed for each detection based on:
  - Consensus: The recurrence of the signal across views.
  - Consistency: The spatial precision (IoU) of the matches.
  - Formula: $\sigma_j = \omega_1 \cdot \text{Consensus} + \omega_2 \cdot \text{Consistency}$ .

3. Key Contributions

Characterization of Failure Modes: The paper identifies semantic fragility (sensitivity to prompt phrasing) and visual inconsistency (sensitivity to image perturbations) as the dominant failure modes for LVLMs in long-tail clinical tasks.
DDL Framework: Introduction of a novel inference-time framework that stabilizes grounding through prompt optimization (DAPE) and perturbation-based verification (V-PUP/RHC) without requiring parameter updates.
Calibration & Reliability: Demonstration that DDL produces a consensus-based reliability score that is well-calibrated with actual localization accuracy, particularly as model scale increases.
Superiority over Baselines: Empirical evidence showing DDL outperforms both manual/automated prompting strategies and supervised fine-tuning (SFT) in rare disease scenarios.

4. Experimental Results

The method was evaluated on two brain MRI benchmarks:

BTD: Common pathologies (Glioma, Meningioma, Pituitary).
NOVA: A large-scale, out-of-distribution benchmark with 281 rare pathology types.

Key Findings:

Performance Gains: On the NOVA benchmark, DDL improved mAP@75 by up to 105% for rare disease cases compared to the vanilla baseline. It consistently outperformed SFT and LoRA, proving that stabilizing inference is more effective than parameter adaptation when data is scarce.
Model Scaling: DDL benefits all model sizes (3B to 72B). Notably, a 32B model with DDL outperformed a zero-shot 72B baseline.
Visual vs. Linguistic Uncertainty: Experiments showed that visual uncertainty (via multi-view probing) is a more reliable indicator of grounding accuracy than linguistic uncertainty (token-level entropy).
Calibration: The correlation between the predicted reliability score ( $\sigma$ ) and actual IoU increases monotonically with model size. For large models (32B+), the reliability score becomes a statistically significant predictor of success ( $r > 0.5$ ), whereas small models show weak calibration.
Robustness: DDL effectively suppresses hallucinations and improves spatial alignment, as visualized in qualitative results where DDL outputs closely match ground truth while vanilla models produce diffuse or incorrect boxes.

5. Significance

Paradigm Shift: The paper argues that for open-world medical imaging (especially rare diseases), test-time optimization is a viable and often superior alternative to weight-based adaptation. It bypasses the need for expensive, disease-specific annotation.
Clinical Trust: By providing a quantifiable reliability score derived from consensus, DDL offers a mechanism for clinicians to trust model outputs, distinguishing stable diagnostic signals from noise.
Scalability: The framework is model-agnostic and can be applied to any frozen LVLM, making it highly scalable for emerging medical AI applications where data collection is a bottleneck.

In conclusion, Dynamic Decision Learning transforms the inference process of LVLMs from a static, fragile pass into a dynamic, self-correcting loop, significantly enhancing the reliability and precision of abnormality grounding in rare and underrepresented medical conditions.