Technical Summary: RadAgent

Problem Statement

While Vision-Language Models (VLMs) have advanced the automated interpretation of medical imaging, particularly Computed Tomography (CT), existing methods often function as "black boxes." They generate final reports without exposing the intermediate reasoning steps, evidence integration, or decision-making logic. This lack of transparency prevents clinicians from inspecting, validating, or refining the AI's output, which is critical in high-stakes diagnostic tasks like chest CT interpretation. Furthermore, current training-free agentic systems rely on static prompts or predefined tool sequences, which may fail to adapt to complex, dynamic clinical workflows or lack the necessary medical knowledge to design optimal diagnostic plans.

Methodology

The authors introduce RadAgent, a tool-using AI agent trained via Reinforcement Learning (RL) to perform stepwise, interpretable chest CT analysis.

Architecture and Workflow

RadAgent operates on a ReAct (Reasoning + Acting) pattern, structured around a clinician-inspired diagnostic checklist. The workflow proceeds as follows:

1. Initialization: Upon receiving a 3D CT volume and a user query, RadAgent invokes a baseline 3D VLM (CT-Chat) to generate an initial report draft.
2. Iterative Reasoning Loop: The agent enters a loop where it:
   • Plans: Decides the next diagnostic action based on the checklist and current findings.
   • Acts: Selects and calls a specific tool from a specialized toolbox via the Model Context Protocol (MCP).
   • Observes: Updates a persistent "scratchpad" with the tool's output.
   • Refines: Integrates new evidence to update its preliminary findings.
3. Termination: Once the agent determines sufficient evidence has been gathered to satisfy the checklist, it synthesizes the accumulated data into a final report.

Toolset

The agent interacts with a toolbox of ten specialized tools, accessible via MCP servers, including:

• Vision-Only Tools: Organ segmentation, effusion segmentation, disease classification (based on CT-CLIP), and CT windowing.
• Vision-Language Tools: 3D VQA (using CT-Chat) and 2D VQA (using Gemma-3-27b-it) for slice-level reasoning.
• Utility Tools: Slice extraction and selection based on segmentation masks.

Training Paradigm

Unlike training-free agents that rely on fixed prompts, RadAgent is trained using Group Relative Policy Optimization (GRPO). The training utilizes a composite reward curriculum designed to balance three objectives:

1. Report Quality: Measured by F1 scores on 18 common pathologies (macro/micro) and an abnormality-specific F1 score derived from a reasoning model.
2. Tool Use Efficiency: Includes rewards for tool call success, tool diversity, and tool graph coherence (ensuring tool outputs are actually used in subsequent steps).
3. Checklist Adherence & Coherence: An LLM-based judge evaluates whether the agent followed the diagnostic checklist and whether the sequence of tool calls was logical.

The reward schedule is dynamic: an early phase encourages exploration of diverse tool sequences, while a later phase prioritizes checklist adherence and sequence coherence.

Key Contributions

1. Transparent, Stepwise Interpretation: RadAgent generates reports accompanied by a fully inspectable trace of intermediate decisions and tool interactions, allowing clinicians to verify how findings were derived.
2. RL-Optimized Tool Use: The system demonstrates that training agents with RL allows for the automatic discovery of effective tool-use strategies, outperforming static, training-free agentic systems.
3. Novel Evaluation Metrics: The study introduces and evaluates Faithfulness and Robustness in the context of medical reporting, metrics previously unmeasured in 3D VLM baselines.

Results

Experiments were conducted on the CT-RATE dataset (internal validation/test) and the RadChestCT dataset (external evaluation).

• Clinical Accuracy: RadAgent significantly outperformed the baseline 3D VLM (CT-Chat).
  • Macro-F1: Improved by 5.8 points (35.4% relative increase) on the CT-RATE test set.
  • Micro-F1: Improved by 5.1 points (18.6% relative increase).
  • Gains were particularly notable for pathologies frequently missed by the baseline.
• Robustness: Under adversarial conditions (injected incorrect hints), RadAgent showed a 24.7 point (41.9% relative) improvement in robustness over CT-Chat (83.7% vs. 58.9%), indicating a reduced susceptibility to misleading prompts.
• Faithfulness: RadAgent achieved a 37.0% faithfulness score, whereas CT-Chat scored 0.0%. This indicates that RadAgent's outputs explicitly reflect the factors (including injected hints) that influenced its decisions, whereas the baseline produced plausible but ungrounded justifications.
• Training-Free Baseline: Even without RL optimization, a training-free version of RadAgent surpassed CT-Chat in macro-F1, suggesting the value of the tool-augmented architecture itself. However, RL provided significant additional gains, particularly in out-of-domain generalization (RadChestCT).

Significance and Claims

The paper posits that RadAgent represents a shift toward transparent and reliable AI for radiology. By structuring interpretation as an explicit, tool-augmented, and iterative reasoning trace, the system addresses the "black box" limitation of current VLMs.

The authors claim that:

• Interpretability is actionable: The ability to inspect the "scratchpad" and tool trace allows clinicians to validate specific findings (e.g., requesting a segmentation of a pleural effusion to visually verify a positive finding).
• RL enables adaptability: Training agents to reason through intermediate steps allows them to acquire domain-specific competencies and learn how to query specialized tools effectively, bridging the gap between general adaptability and specialized precision.
• Future Workflow: The system envisions a "human-in-the-loop" workflow where the agent generates a draft, and the clinician can directly interrogate the underlying evidence within the RadAgent environment.

The authors acknowledge limitations, including the computational cost of hosting multiple GPU-intensive tools and the fact that the agent is optimized for a specific toolset, though they suggest the RL pipeline can be rerun if the toolbox evolves. They also note that while faithfulness improved significantly, the 37.0% score indicates room for further development.