💻 computer science

Counterfactual Anatomy-guided Spatial-Temporal Decoding for Annotation-Free Hallucination Mitigation in Medical VLMs

The paper proposes Counterfactual Anatomy-guided Spatial-Temporal (CAST) decoding, an annotation-free inference framework that mitigates hallucinations in medical vision-language models by automatically identifying causally relevant anatomical regions via counterfactual intervention and applying unified contrastive decoding to enhance spatial grounding and generation dynamics.

Original authors: Yifan Lu, Adinath Dukre, Abhijit Das, Ziyun Zou, Haolin Yang, Yutong Xie, Imran Razzak

Published 2026-08-19

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yifan Lu, Adinath Dukre, Abhijit Das, Ziyun Zou, Haolin Yang, Yutong Xie, Imran Razzak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, a new class of systems known as medical vision-language models has emerged, capable of looking at medical images like X-rays or MRIs and answering questions about them in natural language. These tools hold immense promise for assisting doctors, offering a way to quickly interpret complex scans and explain findings to patients. However, like any powerful tool, they are not perfect. A persistent problem known as "hallucination" plagues these systems, where the model generates fluent, confident-sounding answers that are simply not supported by the image evidence. In a medical context, a confident but incorrect statement about a patient's condition can be dangerous, potentially misleading a clinician or delaying proper care. The challenge for researchers has been to fix this tendency to invent facts without having to retrain the massive models from scratch, a process that is often too expensive and time-consuming.

To address this, a team of researchers has developed a new method called CAST, which acts as a real-time guide for these AI models during the moment they generate an answer. Instead of relying on pre-labeled maps of human anatomy, which are expensive to create and often too broad to be useful, this system teaches the model to find the specific, small area of an image that matters most for a given question. The researchers discovered that by automatically identifying these compact regions and using a two-step correction process, they could significantly reduce the number of false claims the models make. The method works entirely while the model is thinking, requiring no extra training and no human experts to draw boxes around organs beforehand.

The core of the problem lies in how these models currently handle visual information. When asked a question, they often look at the entire image at once, which can lead them to focus on irrelevant details or rely on general knowledge they have memorized rather than what is actually visible. Previous attempts to fix this involved using "ground truth" annotations, where human experts draw precise outlines around the relevant body parts to guide the AI. However, the researchers found that these human-drawn outlines are often too large, covering so much of the image that they dilute the specific visual clues needed to answer the question accurately. It is like trying to find a specific needle in a haystack by pointing to the entire pile of hay; the signal gets lost in the noise.

The CAST framework solves this by first proposing a wide variety of potential anatomical regions using a separate, specialized segmentation tool. It then acts as a selector, testing each candidate region to see which one is truly responsible for the answer. It does this by simulating a "what if" scenario: it temporarily hides or blurs each candidate region and observes how much the model's confidence in its answer drops. If hiding a specific small area causes the model to lose its certainty, that area is identified as the critical piece of evidence. The system specifically looks for compact regions that are just large enough to contain the relevant evidence but small enough to avoid including distracting background information. This process allows the system to automatically discover the precise spot on an X-ray or MRI that holds the key to the question, without needing any human labels.

Once the most relevant region is identified, the system guides the model's generation process through a unified strategy that corrects two types of errors simultaneously. First, it ensures the model is paying attention to the right place. It does this by comparing the model's prediction when it sees the full image against its prediction when the critical region is hidden. By amplifying the difference between these two views, the system forces the model to focus its attention on the specific anatomical evidence rather than guessing based on the rest of the image. Second, it regulates the flow of the answer as it is being written. The system compares the current word being generated with the previous step, suppressing words that seem to be drifting away from the visual evidence and reinforcing those that are firmly grounded in what the model is seeing. This dual approach ensures that the model stays anchored to the image throughout the entire sentence it constructs.

The researchers tested this method on three different medical AI models using two major datasets containing thousands of chest X-rays and other medical scans. The results showed a consistent and significant improvement in accuracy, particularly for questions that required a definitive yes or no answer. In one specific test, the method boosted the accuracy of a leading model on closed-ended questions from roughly 61 percent to over 81 percent. This improvement was achieved without any retraining of the underlying models and without any manual annotations. The study also compared their approach against methods that relied on human-drawn ground truth masks. Surprisingly, the automatically selected, compact regions provided better guidance than the human-drawn ones, which were often too coarse and covered too much area. The data indicated that when the guiding mask covered more than 40 percent of the image, the performance actually dropped, confirming that precision and compactness are far more valuable than broad coverage.

Beyond the numbers, the researchers examined specific cases where the new method succeeded where others failed. In one instance involving a chest X-ray, a standard model and a method using human-drawn boxes both failed to detect a subtle medical device because the guiding area was too large, blurring the specific details of the device's tip. The CAST system, by isolating a tiny, precise region around the device, correctly identified its presence. In another case involving a brain scan, the model needed to identify the body part containing the organ. While other methods incorrectly focused on the organ itself, the new system correctly identified the head as the location by selecting a compact mask that highlighted the surrounding cranial features. These examples demonstrate that the ability to dynamically select the right region is crucial for navigating the nuances of medical imaging.

The entire process is designed to be fast and efficient, fitting seamlessly into the time it takes for a model to generate an answer. The initial proposal of regions is cached, and the selection process involves only a brief evaluation, adding negligible delay to the overall system. This makes the approach practical for real-world deployment, offering a way to make medical AI more reliable and trustworthy without the heavy burden of retraining or the need for vast libraries of expert-labeled data. By teaching the model to find the needle in the haystack on its own, the researchers have provided a practical path toward reducing errors and ensuring that AI-generated medical insights remain firmly grounded in the visual evidence before them.

Technical Summary: Counterfactual Anatomy-guided Spatial-Temporal Decoding (CAST)

1. Problem Statement

Medical Vision-Language Models (Med-VLMs) have demonstrated strong performance in Medical Visual Question Answering (Med-VQA) but remain prone to hallucinations, generating fluent yet clinically unsupported statements that lack grounding in image evidence. In medical contexts, such errors can mislead clinical interpretation and decision-making.

Existing mitigation strategies applied during decoding (e.g., Visual Contrastive Decoding, DoLA, OPERA) are often anatomy-agnostic, applying global corrections that fail to account for the localized nature of medical evidence. While Anatomical Region-Guided Contrastive Decoding (ARCD) attempts to address this by using spatial grounding, it relies heavily on ground-truth (GT) ROI annotations. These annotations are costly, scarce, and often coarse (e.g., large bounding boxes covering entire organs), which can dilute the contrastive signal by removing too much visual information from the unconditional branch. Consequently, there is a need for an annotation-free framework that can automatically discover compact, question-relevant anatomical regions to guide hallucination mitigation.

2. Methodology: The CAST Framework

The authors propose Counterfactual Anatomy-guided Spatial-Temporal (CAST) decoding, a plug-and-play, inference-time framework that requires no model retraining or manual annotations. The pipeline consists of three stages:

Stage 1: Zero-Shot Anatomical Region Proposal

Instead of relying on GT, CAST leverages a text-guided medical segmentation model (MedSAM3) to generate diverse anatomical region candidates.

Multi-Concept Inference: A comprehensive vocabulary of anatomical concepts (thoracic, abdominal, musculoskeletal, vascular) is encoded into a unified prompt.
Parallel Generation: The model's decoder processes this joint embedding to activate parallel queries, generating segmentation masks for multiple structures in a single forward pass.
Filtering: Raw masks undergo a compactness-aware filtering pipeline (area coverage filtering, spatial Non-Maximum Suppression based on IoU, and morphological filtering) to produce a spatially diverse set of $K$ candidate regions.

Stage 2: Counterfactual Region Selection

CAST autonomously selects the single region $m^*$ most causally responsible for the model's reasoning regarding a specific query $q$ . This is framed as a causal discovery problem using counterfactual intervention.

Counterfactual Intervention: For each candidate region $m_k$ , the image is occluded using complementary operators (blur and mean) to disrupt texture and local contrast without introducing artifacts.
Causal Effect (CE) Quantification: The causal effect is measured as the drop in the predictive log-likelihood of a draft answer when the region is occluded:
$CE_o(m_k) = \log p(\hat{y} | x, q) - \log p(\hat{y} | x^{(o,k)}_{occ}, q)$
Selection Objective: The optimal mask is selected by balancing causal relevance with mask compactness and cross-modal consistency:
$S(m_k) = \frac{1}{2} \sum_{o} CE_o(m_k) - \lambda \cdot \text{area}(m_k) - \mu \cdot \text{Var}[\dots]$
This objective explicitly penalizes overly large masks (which provide no localized contrast) and artifacts, ensuring the selection of a compact, question-relevant ROI.

Stage 3: Unified Spatial-Temporal Contrastive Decoding

Once $m^*$ is identified, CAST performs a unified decoding process to correct two orthogonal axes of hallucination: spatial grounding (where the model attends) and temporal generation (what it predicts).

Spatial Contrast (Anatomy-Guided CFG): A dual-branch forward pass is constructed. The unconditional branch suppresses attention to tokens within $m^*$ , while the conditional branch processes the full image. The logits are adjusted using Classifier-Free Guidance (CFG):
$\ell^{(t)}_{CFG} = \log p^{(t)}_u + \gamma \cdot (\log p^{(t)}_c - \log p^{(t)}_u)$
This amplifies tokens dependent on the targeted anatomical region.
Temporal Contrast: To regulate generation dynamics and suppress tokens driven by internal language momentum, CAST applies a step-wise contrast against the previous step's distribution:
$\ell^{(t)}_{final} = w_m \cdot \ell^{(t)}_{CFG} - w_p \cdot \ell^{(t-1)}_{CFG}$
This selectively amplifies tokens newly activated by current visual evidence while suppressing those passively inherited from previous steps.

3. Key Contributions

Annotation-Free ROI Discovery: CAST introduces a method to automatically discover and select compact, question-relevant anatomical regions using counterfactual intervention, eliminating the dependency on expensive ground-truth annotations.
Compactness-Driven Contrast: The authors demonstrate that compact, automatically selected masks provide a cleaner and more effective contrastive signal than coarse ground-truth bounding boxes, which often dilute visual evidence.
Unified Spatial-Temporal Decoding: The framework combines anatomy-guided spatial grounding with step-wise temporal contrast to simultaneously correct attention mechanisms and generation dynamics.
Plug-and-Play Implementation: The entire pipeline operates at inference time across multiple Med-VLMs without requiring retraining.

4. Experimental Results

The framework was evaluated on the SLAKE and MIMIC-CXR datasets using three Med-VLMs: Phi-3.5V-Med, HuatuoGPT-Vision-7B, and LLaVA-Med-7B.

Performance: CAST consistently outperformed strong decoding-time baselines (VCD, DoLA, OPERA) and the GT-dependent ARCD method.
- On Phi-3.5V-Med (SLAKE), CAST improved closed-ended accuracy from 61.42% to 81.49%.
- On MIMIC-CXR, CAST improved both open-ended recall and closed-ended accuracy simultaneously.
Ablation Studies:
- Removing the counterfactual selection (replacing with random masks) degraded overall accuracy by 4.4%.
- Removing the area penalty ( $\lambda=0$ ) caused a 3.4% drop, confirming the necessity of mask compactness.
- Isolating spatial CFG or temporal contrast individually resulted in significant performance drops, validating their orthogonal roles.
Qualitative Analysis: In cases where GT-based methods failed (e.g., due to overly large bounding boxes diluting the signal), CAST successfully isolated compact regions (e.g., a catheter tip or specific craniofacial landmarks) to provide correct diagnoses.

5. Significance and Claims

The paper claims that CAST offers a practical and generalizable solution for improving spatial grounding and reducing hallucinations in Med-VLMs without the need for expert annotations.

Theoretical Insight: The results validate the hypothesis that dynamically selected, compact region masks provide a superior contrastive signal compared to static, coarse ground-truth annotations. The authors argue that GT annotations in benchmarks are often too large, removing too much visual context and weakening the region-specific contrast required for effective decoding.
Practical Impact: By operating entirely at inference time and requiring no retraining, CAST addresses the challenge of improving faithfulness in Med-VLMs without the prohibitive cost of retraining or the scarcity of annotated medical data.
Efficiency: The framework introduces minimal computational overhead, with precomputed segmentation proposals and lightweight counterfactual scoring, making it comparable in latency to existing methods like DoLA.

The authors conclude that CAST achieves state-of-the-art performance across multiple models and datasets, demonstrating that compact, automatically selected regions can effectively guide generation dynamics and reduce hallucinations in medical AI.