Multi-Modal Multi-Agent Reinforcement Learning for Radiology Report Generation: Radiologist-Like Workflow with Clinically Verifiable Rewards

1. Problem Statement

Automated Radiology Report Generation (RRG) aims to alleviate the burden on radiologists by generating diagnostic reports from Chest X-ray (CXR) images. While recent Large Language Models (LLMs) and Vision-Language Models (VLMs) have shown promise, existing approaches face three critical limitations:

• Monolithic Optimization: Most Reinforcement Learning (RL) methods optimize a single, monolithic model rather than a collaborative system.
• Lack of End-to-End Training: Existing "agentic" systems often rely on pre-trained models combined via prompting (training-free) without joint optimization, leading to suboptimal coordination.
• Workflow Misalignment: Current models often fail to mimic the specific, region-by-region diagnostic workflow of human radiologists, resulting in errors in spatial reasoning (e.g., laterality confusion) and lack of detailed anatomical analysis.

The paper argues that to achieve clinical-grade accuracy, the system must jointly train multiple agents within a workflow that mirrors real-world radiologist practices, optimized via clinically verifiable rewards rather than just linguistic similarity.

2. Methodology: MARL-Rad

The authors propose MARL-Rad, a novel Multi-Modal Multi-Agent Reinforcement Learning framework.

A. Agent Architecture

The system mimics the radiologist's workflow by decomposing the task into specialized agents:

1. Region-Specific Agents: Three distinct agents are responsible for specific anatomical regions:
   • Left-Region Agent: Analyzes the left lung, hilar structures, costophrenic angle, and clavicle.
   • Right-Region Agent: Analyzes the right lung, hilar structures, costophrenic angle, and clavicle.
   • Central-Region Agent: Analyzes the cardiac silhouette, mediastinum, aortic arch, trachea, and spine.
2. Global Integrating Agent: Receives the outputs from the three region-specific agents, synthesizes their diagnoses, and generates the final comprehensive report (Findings and Impression).

B. Reinforcement Learning Algorithm (MA-GSPO)

The framework extends Group Sequence Policy Optimization (GSPO) to a multi-agent setting:

• Joint Rollouts: For a given query (image), the system samples a group of joint trajectories where all agents generate outputs sequentially.
• Group-Relative Advantage: Rewards are calculated based on the group's performance relative to the mean, reducing variance.
• On-Policy Optimization: Unlike previous off-policy or post-hoc agentization methods, MARL-Rad performs on-policy optimization of the entire agent system simultaneously. This ensures that agents learn to collaborate effectively within the specific workflow.

C. Clinically Verifiable Rewards

Instead of relying solely on natural language generation (NLG) metrics (like BLEU), the reward function is based on clinically verifiable metrics:

• CheXbert Accuracy: Measures the correctness of predicted clinical findings (e.g., presence of pneumonia).
• RadGraph F1: Evaluates the factual and relational consistency of clinical entities and their relationships.
• ROUGE-L: Included to maintain linguistic naturalness.
  The final reward is the unweighted sum of these components, driving the model toward clinical accuracy rather than just textual fluency.

3. Key Contributions

1. End-to-End Multi-Agent Optimization: The first framework to jointly optimize a multi-agent system for RRG using on-policy RL, moving beyond training-free prompting or single-agent RL.
2. Radiologist-Like Workflow: The architecture explicitly decomposes the task into anatomical regions, addressing the "global image parsing" limitation of standard VLMs and improving spatial reasoning.
3. State-of-the-Art Clinical Efficacy: The method achieves superior performance on clinical efficacy (CE) metrics compared to all prior state-of-the-art (SOTA) methods.
4. Enhanced Laterality Consistency: The specialized left/right agents significantly reduce errors in identifying left vs. right pathologies, a critical safety requirement in radiology.

4. Experimental Results

The method was evaluated on two major datasets: MIMIC-CXR and IU X-ray.

• Performance Metrics:
  • Clinical Efficacy (CE): MARL-Rad achieved SOTA scores in RadGraph F1, CheXbert F1, and GREEN scores on both datasets. For example, on MIMIC-CXR, it achieved a CheXbert F1 of 0.556 (Findings + Impression), outperforming the previous best (MedGemma at 0.523).
  • Generalization: Despite being trained only on MIMIC-CXR, the model demonstrated strong generalization on the IU X-ray dataset.
• Ablation Studies:
  • Removing the multi-agent structure (using a single agent with RL) resulted in lower CE scores.
  • Using the multi-agent workflow without RL (training-free) performed worse than the vanilla base model, proving that joint RL optimization is essential for agent coordination.
• Laterality Analysis:
  • A specific analysis of "Left" and "Right" entities in RadGraph showed MARL-Rad significantly outperformed single-agent RL baselines, confirming that region-specific agents effectively capture spatial relationships.
• Case Studies:
  • Visual comparisons showed that MARL-Rad produces reports with detailed, region-specific observations that are correctly synthesized, whereas single-agent models often produce vague summaries or omit specific regional findings.

5. Significance and Conclusion

This paper represents a paradigm shift in medical AI report generation. By moving from monolithic models to collaborative, workflow-aligned multi-agent systems, the authors demonstrate that:

• Clinical Accuracy > Linguistic Fluency: Optimizing for clinically verifiable rewards yields reports that are more useful to physicians than those optimized for BLEU/ROUGE scores.
• Workflow Alignment is Crucial: Mimicking the human cognitive process (examining regions sequentially) improves the model's ability to handle complex spatial reasoning tasks like laterality.
• Scalability: While focused on CXR, the MARL framework is generalizable to other medical modalities (e.g., ECG, Echocardiography) and real-world tasks requiring structured, multi-step reasoning.

The work establishes a new benchmark for automated radiology reporting, prioritizing safety, accuracy, and the structural integrity of diagnostic reasoning over simple text generation.