💻 computer science

Synergistic Perception-Reasoning Governance: Grounding Medical MLLMs with Verifiable Anatomical Evidence

This paper proposes Synergistic Perception-Reasoning Governance (SPRG), a training-free framework that mitigates hallucinations in medical multimodal large language models by injecting verifiable anatomical evidence through ROI-guided visual modulation and coordinate-to-token semantic anchoring, achieving significant improvements in accuracy and hallucination reduction across diverse benchmarks.

Original authors: Rui Hao, Qiankun Li, Junyuan Mao, Linghao Meng, Dirui Xie, Dayu Tan, Zhigang Zeng

Published 2026-07-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rui Hao, Qiankun Li, Junyuan Mao, Linghao Meng, Dirui Xie, Dayu Tan, Zhigang Zeng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a brilliant but sometimes overconfident medical student who is great at talking but occasionally makes things up when looking at an X-ray. This student might say, "I see a broken bone here," even though the bone looks perfectly fine, simply because they are trying to sound smart or because they are distracted by the background noise of the image. In the world of Artificial Intelligence, this is called a hallucination.

This paper introduces a new "supervisor" system designed to stop these medical AI models from making things up, without needing to retrain them or teach them new lessons. Here is how it works, broken down into simple concepts:

The Problem: The "Confident but Wrong" AI

Current medical AI models (called Multimodal Large Language Models) are like that student. They can look at an X-ray and write a report, but sometimes they confidently describe diseases or body parts that aren't actually there. Existing solutions to fix this are often like hiring a whole new team of teachers (retraining) or building a massive library of facts to check against (external databases), which is expensive and complicated.

The Solution: The "Evidence Injection" Framework

The authors propose a clever, training-free method. Think of it as giving the AI a pair of "truth goggles" and a "fact-checking notepad" right before it answers a question. They call this Synergistic Perception-Reasoning Governance.

Here are the three main tricks they use:

1. The "Spotlight" (Vision-Side Recalibration)

Imagine the AI is looking at an X-ray, but its vision is a bit blurry, and it's getting distracted by the edges of the photo or the background.

What they do: They use a tool called MedSAM (a smart segmentation tool) to automatically draw a box around the specific body part the doctor is asking about (like the "left atrium" of the heart).
The Analogy: It's like shining a bright spotlight only on that specific body part and turning the rest of the room (the background noise) into darkness.
The Result: The AI is forced to pay attention only to the evidence inside the box. If the box says "heart," the AI can't hallucinate a "broken leg" because the spotlight isn't there.

2. The "Fact Card" (Text-Side Evidence Injection)

Sometimes, just shining a light isn't enough; the AI needs a reminder of the facts.

What they do: They take the coordinates of that box (e.g., "This box is at position X, size Y, and it covers 5% of the image") and turn them into a simple text sentence. They paste this sentence right into the AI's question.
The Analogy: It's like handing the student a cheat sheet that says, "Remember, the heart is located right here in the image."
The Result: The AI's reasoning is "anchored" to this physical evidence. It can't wander off into fantasy because the text explicitly tells it where the evidence is.

3. The "Smart Traffic Cop" (Task-Aware Dynamic Router)

Not every medical question is the same. Some ask "Where is the heart?" (needs the spotlight), while others ask "Is there fluid?" (needs the fact card), and some are complex and need both.

What they do: They built a tiny, fast "traffic cop" that reads the question first.
The Analogy: If the question is about location, the cop waves the AI to use the Spotlight. If the question is about measurements or symptoms, the cop hands over the Fact Card. If it's a complex radiology report, the cop says, "Use both!"
The Result: The AI gets exactly the right tool for the specific job, balancing accuracy with natural-sounding language.

The Results: Less Lying, More Truth

The researchers tested this system on several different AI models and medical datasets (like chest X-rays).

Accuracy: They found that the AI got the right answers about 6% more often on specific medical questions.
Hallucinations: They reduced the number of made-up facts by about 35%.
Versatility: It worked well across different types of AI models, proving it's a flexible fix.

In a Nutshell

Instead of trying to rebuild the AI's brain, the authors simply gave it verifiable evidence (the "spotlight" and the "fact card") right before it speaks. By forcing the AI to look at the actual evidence and reminding it of the facts, they stopped it from confidently making things up, making it much safer and more reliable for medical use.

Technical Summary: Synergistic Perception-Reasoning Governance (SPRG)

Problem Statement
Multimodal Large Language Models (MLLMs) demonstrate significant potential for clinical Visual Question Answering (VQA) and radiology report generation. However, their deployment in high-stakes medical settings is hindered by "hallucinations," specifically visual misinterpretations where models generate fluent conclusions that contradict or lack support from imaging evidence. Existing mitigation strategies often rely on additional training, external knowledge bases, or complex multi-stage post-hoc verification. These approaches increase computational costs, pipeline complexity, and often fail to generalize across different models and clinical tasks. Furthermore, many current methods rely on internal statistical calibration without explicit anatomical grounding, limiting their robustness.

Methodology
The authors propose Synergistic Perception-Reasoning Governance (SPRG), a holistic, training-free framework designed to mitigate hallucinations at inference time through dual-side evidence injection. The framework operates on three main components:

Anatomical Evidence Extraction:
The system utilizes MedSAM (Segment Anything Model adapted for medical imaging) to extract verifiable anatomical evidence. Given a candidate bounding box set, MedSAM generates segmentation masks. These are processed into a deterministic Region of Interest (ROI) prior set, $\mathcal{R}$ , consisting of segmentation masks, bounding boxes, and anatomical semantic labels derived from the clinical context.
Perception-Reasoning Intervention Mechanism:
The framework intervenes in both the visual and textual trajectories of the MLLM:
- Text-side Evidence Injection: The ROI priors are converted into structured region descriptors (anatomical labels, coordinates, and dimensions) and serialized into discrete semantic tokens. These tokens are injected directly into the textual prompt ( $\tilde{q}$ ) before the model generates an answer. This anchors the textual reasoning trajectory with verifiable external memory.
- Vision-side Activation Recalibration: The system computes the spatial coverage density of the ROI masks over the visual patch tokens. A soft-mask is applied to the visual features ( $\tilde{V}$ ) to modulate activations. Features corresponding to non-ROI areas are attenuated (via a factor $\beta$ ), while ROI-aligned features are preserved or enhanced (via a factor $\alpha$ ). This suppresses background noise and out-of-distribution structural collapse, effectively "excising" spurious visual pathways.
- Joint Modulation: Both strategies can be applied simultaneously to enforce complementary regularization.
Task-Aware Dynamic Router:
Recognizing that different clinical tasks have varying sensitivities to visual versus textual constraints, the authors introduce a lightweight, zero-overhead router. Based on a zero-shot classification of the input question into four categories—Anatomy (CA), Measurement (CM), Symptom (CS), and Radiology features (CR)—the router dynamically selects the optimal intervention:
- Anatomy: Vision-side recalibration (spatial localization).
- Measurement/Symptom: Text-side injection (precise semantic anchoring).
- Radiology features: Joint strategy (complex cross-modal reasoning).

Key Contributions

Novel Framework: Introduction of a training-free framework that mitigates hallucinations by injecting dual-side verifiable anatomical evidence, bridging raw visual perception and clinical reasoning.
Intervention Mechanism: Development of a mechanism that recalibrates visual perception via ROI-guided activation modulation and anchors textual reasoning via discrete semantic tokens derived from anatomical coordinates.
Dynamic Routing: Proposal of a task-aware router that adjusts intervention strategies based on task semantics, balancing perceptual grounding and linguistic fluency.
Comprehensive Evaluation: Establishment of a rigorous cross-model, cross-task evaluation involving 2 tasks, 5 datasets, and 6 MLLMs (including LLaVA-1.5, LLaVA-Med, Qwen3-VL, and InternVL-3.5).

Results
Extensive experiments on MM-VisHal (SLAKE, VQA-RAD) and CXR-VisHal (IU-Xray, MIMIC-CXR) datasets demonstrate the framework's efficacy:

Close-ended Tasks: The framework consistently outperforms baselines across all tested MLLMs. The dynamic routing strategy achieved the highest overall accuracy improvements, with gains of +3.2% on MM-VisHal and +2.4% on CXR-VisHal. Specific strategies showed domain-specific strengths: text-side injection dominated overall accuracy, vision-side constraints excelled in anatomy-related questions, and the joint strategy performed best for radiology features.
Open-ended Tasks: On the MIMIC-CXR report generation task, the vision-side intervention achieved the lowest hallucination rate (CHAIR score of 12.76%), significantly reducing visual misinterpretations. The text-side intervention maintained high recall and improved narrative quality (RaTEScore). The joint strategy offered a strong balance between hallucination suppression and report completeness.
Ablation Studies: Results confirmed that improvements rely on the specific spatial grounding of ROI coordinates (random bounding boxes degraded performance) and that the optimal visual weighting configuration is $(\alpha=1.0, \beta=0.1)$ .

Significance
The paper claims that SPRG provides a practical, training-free path toward visually grounded and controllable generation in clinical workflows. By leveraging verifiable anatomical evidence extracted via MedSAM, the method addresses the critical barrier of evidence-inconsistent outputs without requiring model retraining or complex external retrieval systems. The authors emphasize that their approach offers superior robustness over existing baselines, improving close-ended reasoning accuracy by up to ~6% and reducing open-ended hallucinations by ~35%, thereby enhancing the trustworthiness of MLLMs in high-stakes medical decision-making.