💬 NLP

RadAgents: Multimodal Agentic Reasoning for Chest X-ray Interpretation with Radiologist-like Workflows

RadAgents is a novel multi-agent framework that enhances chest X-ray interpretation by integrating clinical priors, task-aware multimodal reasoning, and grounding mechanisms to emulate radiologist workflows, thereby producing clinically aligned, visually grounded, and verifiable diagnostic outputs.

Original authors: Kai Zhang, Corey D Barrett, Jangwon Kim, Lichao Sun, Tara Taghavi, Krishnaram Kenthapadi

Published 2026-04-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kai Zhang, Corey D Barrett, Jangwon Kim, Lichao Sun, Tara Taghavi, Krishnaram Kenthapadi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a complex mystery: What is wrong with a patient's chest based on an X-ray?

For decades, AI has tried to play the role of this detective. But most current AI detectives work like a student who glances at a photo once, memorizes the general vibe, and then writes a report without ever looking back at the photo. They might miss a tiny crack in a rib or confuse a shadow for a tumor because they didn't take the time to measure things or compare them to other cases.

Enter RadAgents, a new AI system designed to think and work exactly like a real human radiologist.

Here is how it works, broken down into simple concepts:

1. The "Specialized Squad" (Instead of One Generalist)

Most AI systems are like a single person trying to do everything: cook dinner, fix the car, and diagnose an illness all at once. They get overwhelmed.

RadAgents is different. It is a team of seven specialized experts working together, just like a hospital team:

The Airway Agent: Only looks at the windpipe and breathing tubes.
The Breathing Agent: Only scans the lungs for pneumonia or fluid.
The Circulation Agent: Only checks the heart size and blood vessels.
The Diaphragm Agent: Checks the muscle below the lungs.
The "Everything Else" Agent: Looks for broken bones, tubes, or foreign objects.
The Orchestrator (The Manager): This is the boss. When you ask a question, the Manager figures out which experts are needed and sends them their specific tasks.
The Synthesizer (The Editor): This agent collects all the reports from the experts, checks if they agree, and writes the final diagnosis.

The Analogy: Imagine a sports team. You don't ask the goalkeeper to run the offense. You ask the striker to score, the defender to block, and the coach to direct the play. RadAgents does exactly this for X-rays.

2. The "Tool Belt" (Not Just Guessing)

Old AI models often just "guess" based on patterns they saw in training. RadAgents, however, carries a digital tool belt.

If the team needs to know if the heart is too big, they don't guess. They use a ruler tool to measure the heart and the chest width. If they need to find a specific shadow, they use a highlighter tool to draw a box around it. If they need to calculate a ratio, they use a calculator.

This means their answers are based on hard evidence (measurements and visual checks) rather than just a "feeling."

3. The "Second Opinion" (When Tools Disagree)

Sometimes, one tool might say "There is fluid here," and another might say "No, that's just a shadow." In the past, AI would get confused or pick one at random.

RadAgents has a clever trick called V-RAG (Visual Retrieval-Augmented Generation).

The Analogy: Imagine a junior doctor is unsure about a diagnosis. They don't just guess; they walk over to the library, pull out three similar past cases from the files, and compare them. "Hey, in Case A and Case B, this shadow meant fluid. In Case C, it was normal. Let's go with the majority."
RadAgents does this instantly. If the tools disagree, it searches its database for similar X-rays to see what the "correct" answer usually is, resolving the conflict before giving a final answer.

4. The "Audit Trail" (No Black Boxes)

Current AI is often a "black box": you put an image in, and a report comes out, but you have no idea how it got there. If it's wrong, you can't tell why.

RadAgents keeps a detailed logbook of every single step.

"Step 1: Measured the heart."
"Step 2: The heart was 10cm wide."
"Step 3: Compared to the chest width."
"Step 4: Calculated the ratio."
"Step 5: Found a conflict, checked 3 past cases."
"Conclusion: Heart is enlarged."

This makes the system trustworthy because a human doctor can look at the logbook, see the math, and verify the work.

Why Does This Matter?

The paper shows that RadAgents is smarter and more accurate than current top-tier AI models, even though it uses smaller, cheaper computer brains.

It's like upgrading from a calculator that just gives you a number, to a team of accountants who check the math, compare it to last year's taxes, and explain exactly where the error is.
It reduces the risk of missing a diagnosis.
It works faster by having experts work in parallel (at the same time) rather than one by one.
It builds trust by showing its work.

In short, RadAgents isn't just trying to "read" an X-ray; it's trying to think like a doctor, using a structured workflow, specialized tools, and a second opinion to ensure the patient gets the best care.

1. Problem Statement

Current AI systems for Chest X-ray (CXR) interpretation face three critical limitations that hinder their clinical adoption:

Lack of Clinical Interpretability: Prevailing methods often rely on "encode-once, text-only" paradigms where a visual encoder processes an image once, and subsequent reasoning occurs purely in text. This decouples reasoning from evolving visual evidence, leading to failures in tasks requiring iterative re-inspection, precise measurements, or cross-view comparisons.
Insufficient Multimodal Fusion: Existing systems often generate text-only rationales that are not visually grounded. They fail to effectively fuse multimodal evidence, resulting in hallucinations or unverified claims.
Absence of Conflict Resolution: Agentic systems that use external tools often produce inconsistent outputs across different tools (e.g., a segmentation model vs. a classification model) without principled mechanisms to detect or resolve these discrepancies.

The paper argues that radiologists do not reason linearly; they follow structured, interleaved workflows involving image inspection, measurement, and hypothesis refinement. Current Large Multimodal Models (LMMs) fail to replicate this traceable, auditable process.

2. Methodology: The RadAgents Framework

RadAgents is a multi-agent system designed to encode radiologist-style workflows into a modular, auditable pipeline. It utilizes open-source, lightweight Vision-Language Models (VLMs) (specifically Qwen3-VL-Instruct) as the core engines.

A. Architecture

The system consists of seven specialized agents:

Orchestrator: Analyzes user queries, extracts clinical intent, and dispatches tasks to specific sub-agents. It uses a hybrid retrieval approach (BM25 + dense embeddings) to match queries with predefined clinical workflows.
Five Sub-Agents (ABCDE Scheme): These agents implement the standard radiologic review scheme:
- Airway: Assesses tracheal patency and alignment.
- Breathing: Surveys lungs/pleura for opacities, nodules, etc.
- Circulation: Evaluates cardiac silhouette and vessels (e.g., Cardiothoracic Ratio).
- Diaphragm: Assesses diaphragm integrity and subdiaphragmatic air.
- Everything-else: Identifies chest wall, fractures, and foreign bodies.
- Mechanism: Each sub-agent follows a "Reason + Act" (ReAct) paradigm, executing specific skill chains (tool sequences) defined in system prompts. If a step fails, the agent triggers local re-planning.
Synthesizer: Aggregates outputs from sub-agents, performs contextual verification, and resolves cross-tool conflicts.

B. Key Technical Components

Tool Set Integration: RadAgents integrates a comprehensive suite of tools including:
- Segmentation: CXAS (anatomy) and BiomedParser (pathology).
- Grounding: MAIRA-2 for localizing text descriptions to bounding boxes.
- Measurement: Custom Python scripts for geometric calculations (e.g., CTR, carinal angle).
- VQA & Classification: MedGemma, CheXagent, and DenseNet-121.
Visual Retrieval-Augmented Generation (V-RAG): To resolve conflicts between tools (e.g., when a segmentation mask suggests one finding but a classifier suggests another), the Synthesizer retrieves the top- $k$ clinically similar CXR studies from a database (MIMIC-CXR) using image embeddings (Rad-DINO). These exemplars serve as reference standards to adjudicate discrepancies.
Short-Term Memory: A shared memory cache stores patient demographics, metadata, and intermediate tool outputs to prevent redundant computation and ensure consistency across multi-step analyses.
Workflow Modes: The system supports Workflow-Guided mode (for standard guideline tasks) and Workflow-Free mode (ReAct-style for open-ended queries).

3. Key Contributions

Formalization of Radiologist Workflows: The paper formalizes the interleaved, structured review process of radiologists into a multi-agent system that explicitly interleaves visual evidence, measurements, and textual reasoning.
Traceable, Conflict-Resolving Architecture: It introduces a novel architecture combining an Orchestrator, specialized sub-agents, and a Synthesizer with V-RAG. This provides step-level traceability and principled handling of cross-tool inconsistencies, significantly improving alignment with clinical practice.
Efficiency and Accessibility: The authors demonstrate that a small, open-source model (8B parameters) paired with structured workflows and tool integration can match or surpass massive proprietary models (like GPT-4o) and specialized CXR models, suggesting that trustworthy medical AI does not strictly require massive proprietary LLMs.

4. Experimental Results

The authors evaluated RadAgents on three challenging benchmarks: ChestAgentBench, CheXbench, and MIMIC-CXR (report generation).

ChestAgentBench: RadAgents achieved the highest accuracy across all 7 categories (Overall: 73.6%), outperforming CheXagent (39.5%), GPT-4o (56.4%), and Qwen3-VL with ReAct (61.3%). The largest gains were seen in "Diagnosis" and "Characterization" tasks.
CheXbench: RadAgents achieved an overall accuracy of 74.6%, surpassing CheXagent (64.7%) and GPT-4o (63.5%). It showed particular strength in fine-grained visual understanding (SLAKE: 89.4%).
MIMIC-CXR Report Generation: Using the GREEN score (an LLM-as-a-judge metric), RadAgents scored 51.4, significantly outperforming GPT-4o (34.2) and CheXagent (23.6).
Ablation Studies:
- V-RAG: Removing visual retrieval dropped performance by ~6–7 percentage points, highlighting the importance of external evidence for resolving conflicts.
- Model Scaling: Increasing the Synthesizer model size from 4B to 30B significantly improved conflict resolution rates (from 26.3% to 60.5%), whereas increasing the Orchestrator size had a more modest effect on task dispatch.
- Efficiency: While multi-agent systems incur higher compute costs than single-agent baselines, parallel execution of sub-agents reduced wall-clock latency by 30–40%, making it viable for asynchronous clinical workflows.

5. Significance

RadAgents represents a paradigm shift from "black-box" end-to-end medical AI to transparent, workflow-aligned agentic systems.

Clinical Trust: By mimicking the radiologist's structured review and providing explicit logs of intermediate artifacts (masks, measurements, retrieved cases), the system offers the traceability required for clinical auditing and error analysis.
Robustness: The V-RAG mechanism allows the system to self-correct when individual tools fail or disagree, a critical feature for high-stakes medical diagnosis.
Accessibility: The success of the 8B parameter model suggests that high-performance medical AI can be deployed on-premise with accessible hardware, reducing reliance on expensive proprietary APIs.

In conclusion, RadAgents demonstrates that combining structured clinical workflows, specialized tool use, and visual retrieval can create a robust, interpretable, and highly accurate system for chest X-ray interpretation, effectively bridging the gap between current AI capabilities and real-world clinical practice.