💻 computer science

MedScribe: Clinically Grounded CT Reporting through Agentic Workflows

MedScribe is a hypothesis-driven framework that enhances CT report accuracy and grounding by reformulating generation as an iterative, agentic process where a large language model dynamically invokes diagnostic tools to accumulate quantitative evidence before synthesis, outperforming existing vision-language models without task-specific fine-tuning.

Original authors: Giuseppe A. Orlando, Paolo Papotti, Maria A. Zuluaga, Olivier Humbert, Marco Lorenzi

Published 2026-05-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Giuseppe A. Orlando, Paolo Papotti, Maria A. Zuluaga, Olivier Humbert, Marco Lorenzi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to write a detailed report about a complex 3D object, like a giant, multi-layered cake, but you can only look at it through a tiny keyhole. You have to guess what's inside the layers you can't see. This is essentially the problem current AI models face when trying to write medical reports for 3D CT scans (which are like 3D X-rays of the body). They often try to squish the entire 3D scan into a single, tiny summary, which leads to them "hallucinating" or making up details they can't actually see.

The paper introduces MedScribe, a new way of doing this that acts less like a guesser and more like a detective.

The Old Way: The "One-Shot" Guess

Think of traditional AI models as a student taking a test who is handed a massive textbook, told to read the whole thing in one second, and then immediately asked to write an essay. Because they can't process the whole book at once, they have to compress it into a tiny note. When they write the essay, they might confidently say, "There is a red spot on page 42," even though they never actually saw page 42. In medical terms, this leads to reports that sound fluent but contain fake findings or miss real ones.

The New Way: MedScribe (The Detective)

MedScribe changes the game. Instead of trying to swallow the whole 3D scan at once, it uses a step-by-step investigation process.

Here is how it works, using a simple analogy:

1. The Detective (The AI Brain)
MedScribe uses a smart AI (a Large Language Model) as the "Detective." This detective doesn't just stare at the scan; it has a specific job: to figure out what's wrong by asking specific questions.

2. The Specialized Tools (The Magnifying Glass & Ruler)
The detective doesn't have "eyes" that can see everything perfectly. Instead, it has a toolbox of specialized instruments.

If the detective suspects a fluid buildup in the lungs, it doesn't guess. It calls a "Fluid Tool" that acts like a precise ruler, measuring exactly how much fluid is there and where it is located.
If it suspects a hard spot (calcification), it calls a "Density Tool" that measures the hardness of that specific spot.
These tools give the detective hard numbers (like "53mm thick" or "located on the left side") rather than vague guesses.

3. The Reference Library (The Case Files)
Once the tools give the detective some numbers, the detective doesn't just make up a story. It runs to a massive library of past case files (a database of real CT scans and their reports).

It asks the library: "I have a fluid measurement of 53mm on the left. What did real doctors say in similar cases?"
The library hands back snippets of real reports that match those specific numbers. This ensures the detective is grounded in reality, not just making things up.

4. The Report (The Final Conclusion)
Only after gathering these hard numbers and checking them against real past cases does the detective write the final report. Because it built the report on evidence it actually collected, the report is much more accurate and less likely to contain "hallucinations."

Why This Matters (According to the Paper)

The authors tested this "Detective" approach against other top-tier AI models (like Gemini and MedGemma) using two large databases of real chest CT scans.

Better Accuracy: MedScribe was much better at correctly identifying specific problems (like fluid on the left vs. right side of the lungs) compared to the other models.
No "Fake" Findings: Because the AI had to use a tool to measure something before writing about it, it stopped making up findings that weren't there.
No Re-Training Needed: The best part is that MedScribe didn't need to be "re-trained" on millions of new examples to learn this behavior. It just needed to be given the tools and the library, and it figured out how to use them on its own.

The Catch (Limitations Mentioned)

The paper is honest about one weakness: The "Detective" relies on the "Tools" to measure things. If the tool that measures the lungs makes a mistake (like drawing the outline of the lung too big), the detective gets bad numbers, and the final report suffers. However, the paper notes that this is actually a good thing because it makes the AI's mistakes traceable. You know exactly where the process went wrong (the measurement step), rather than the AI just being mysteriously wrong.

In Summary

MedScribe stops trying to be a "magic oracle" that sees everything at once. Instead, it acts like a methodical doctor: it forms a hypothesis, uses a tool to measure the evidence, checks that evidence against past cases, and then writes the report. This makes the AI's reasoning transparent, grounded in facts, and much more reliable for medical imaging.

Technical Summary: MedScribe – Clinically Grounded CT Reporting Through Agentic Workflows

Problem Statement

Current Medical Vision-Language Models (VLMs) for radiology report generation face significant limitations when applied to 3D Computed Tomography (CT) imaging. Existing approaches typically rely on global embedding compression of volumetric data, where a fixed embedding is generated from the entire 3D volume before text generation. This "single-pass" encoding creates an information bottleneck that often corrupts subtle pathological patterns and leads to hallucinated findings. Furthermore, these models lack mechanisms to explicitly verify which anatomical regions drive a generated finding, resulting in reports that lack precise correspondence with underlying pathological evidence and suffer from limited anatomical grounding. While recent datasets like CT-RATE have enabled the development of 3D-native models (e.g., CT-CHAT, CT2Rep), they still inherit the constraints of ViT-based encoders and cannot explicitly validate clinical hypotheses against volumetric data.

Methodology: MedScribe Framework

MedScribe reformulates 3D radiology report generation not as a single-pass encoding task, but as an iterative, hypothesis-driven evidence acquisition process. Inspired by the Reasoning and Acting (ReAct) framework, MedScribe operates as a tool-augmented agent that decouples linguistic reasoning from visual perception.

Core Workflow

Sequential Decision Process: Instead of compressing the entire CT scan ( $V$ ) into a fixed embedding, MedScribe treats report generation as a sequence of reasoning steps ( $T$ ). The agent state at step $t$ is defined as $s_t = (q, E_t)$ , where $q$ is the user query and $E_t$ is the accumulated evidence.
Tool Invocation: The Large Language Model (LLM) acts as a central reasoning agent. At each step, it selects an action $a_t$ $a_{t}$ from a set of pathology-specific diagnostic tools ( $\mathcal{A}$ $A$ ). These tools are not learned parameters but external modules that extract localized volumetric features.
- Feature Extraction: Tools utilize segmentation models (e.g., TotalSegmentator) and radiomics analysis to produce structured descriptors ( $f_t$ ), including volumetric measurements, intensity statistics (Hounsfield Units), and anatomical coordinates (laterality, orientation).
- Local Access: The CT volume is accessed only locally through these tool calls, avoiding global downsampling.
Multidimensional RAG Retrieval: The structured feature vectors $f_t$ $f_{t}$ are used to query a multidimensional Retrieval-Augmented Generation (RAG) space. This space is grounded in quantitative radiomics descriptors and textual snippets from the CT-RATE reference dataset.
- The system retrieves the top $k$ similar reference cases based on feature proximity (L2 distance).
- Retrieved snippets provide grounded visual-to-textual mappings, allowing the LLM to verify hypotheses against real-world examples.
Evidence Accumulation & Synthesis: The evidence set $E_{t+1}$ is updated with the new features and retrieved text. The loop continues until the LLM determines sufficient evidence is accumulated (triggering a STOP action). Finally, the LLM synthesizes the final report $r = g(q, E_T)$ .

Operational Modes

The framework supports two distinct modes:

MedScribe-Template: The agent starts with a neutral boilerplate report and iteratively builds evidence using tools.
MedScribe-VLM: The agent acts as an automated reviewer, initializing the evidence set with a baseline VLM's draft and then invoking tools to verify or correct findings.

Key Contributions

Agentic Framework for 3D Reporting: A novel architecture that decouples linguistic reasoning from visual perception via iterative tool invocation, moving away from static end-to-end encoders.
Quantitative RAG Space: A multidimensional retrieval space conditioned on quantitative anatomical descriptors (radiomics) rather than static text or visual embeddings, ensuring factual grounding.
Zero-Shot Performance: The framework achieves state-of-the-art results without task-specific fine-tuning of the underlying LLM, relying instead on the reasoning capabilities of frontier models (e.g., Llama 3.1-8B) and structured tool outputs.

Experimental Results

Experiments were conducted on the CT-RATE (primary) and RadChestCT (generalizability) datasets, comparing MedScribe against state-of-the-art VLMs including CT-CHAT, Gemini 2.5 Flash, MedGemma, and VILA-M3.

Clinical Accuracy: MedScribe demonstrated superior performance in pathology classification (F1 scores) across nearly all 18 evaluated pathologies compared to baseline VLMs. The "MedScribe-Template" configuration consistently outperformed all baselines.
Factual Grounding: The framework significantly improved fine-grained anatomical reasoning, particularly in laterality prediction (left/right) for pleural effusions, a task where baseline VLMs struggled.
Ablation Studies:
- Removing the RAG component (relying only on raw features) caused a significant drop in precision (0.18 vs. 0.36), highlighting the necessity of retrieval for reducing false positives.
- Removing the agentic loop (providing all tools at once) resulted in lower performance, confirming the value of iterative reasoning.
- Replacing the LLM synthesis with rule-based classifiers maintained classification metrics but failed in text generation and laterality tasks, underscoring the need for LLM synthesis.
Sensitivity: Performance degradation was observed in cases with unreliable segmentation (oversegmentation), confirming that the pipeline's interpretability allows errors to be localized to specific anatomical regions rather than causing global failure.

Significance and Claims

The paper posits that MedScribe offers a clinically motivated inductive bias for structured reporting by enforcing a transparent chain of evidence. Unlike passive VLMs that compress data into opaque embeddings, MedScribe explicitly accumulates quantitative evidence before synthesis.

The authors claim that this approach:

Reduces Hallucinations: By grounding findings in extracted numerical descriptors and retrieved reference cases, the framework minimizes unsupported claims.
Enhances Interpretability: The failure modes are traceable to specific components (e.g., segmentation errors or tool limitations), unlike the "black box" nature of end-to-end models.
Scalability: The framework is extensible; performance gains are driven by the quality of the tool library and the reasoning capabilities of the LLM, suggesting that adding tools for currently under-represented pathologies (e.g., interstitial lung disease) would directly improve coverage.

The paper concludes that hypothesis-driven reasoning, where the model actively queries volumetric data for specific evidence, represents a necessary paradigm shift for reliable 3D medical image reporting, achieving state-of-the-art results without the need for extensive task-specific fine-tuning.