UltrasoundAgents: Hierarchical Multi-Agent Evidence-Chain Reasoning for Breast Ultrasound Diagnosis

1. Problem Statement

Breast ultrasound (BUS) diagnosis follows a specific clinical workflow: clinicians first localize a suspicious lesion globally, then examine fine-grained signs (e.g., morphology, echo patterns), and finally integrate this evidence to assign a BI-RADS category and determine malignancy.

Existing computer-aided diagnosis (CAD) systems face three main limitations:

• Lack of Traceability: Most methods use end-to-end prediction or weakly grounded multi-stage pipelines that fail to provide a clear "evidence chain" explaining how local cues lead to a final diagnosis.
• Weak Interpretability: Generated evidence is often not explicit enough to support clinical auditability or review.
• Training Instability in Hierarchical Models: While hierarchical multi-agent approaches (separating localization from reasoning) are promising, they suffer from error propagation (localization errors corrupt downstream perception), difficult credit assignment, and sparse rewards when trained end-to-end.

2. Methodology: UltrasoundAgents Framework

The authors propose UltrasoundAgents, a hierarchical multi-agent framework that explicitly decouples global localization from fine-grained attribute perception, mimicking the clinical "coarse-to-fine" workflow.

A. Architecture

The system consists of two agents interacting via a Crop-and-Zoom mechanism:

1. Main Agent ($A_M$):
   • Role: Performs global scanning of the full ultrasound image to localize the lesion (predicting a bounding box $b$) and integrates evidence for the final diagnosis.
   • Output: Predicts the malignancy status ($y_{mal}$) and BI-RADS category ($y_{bi}$).
   • Action: Triggers a crop-and-zoom operation to generate a local lesion view ($I_{crop}$) for the sub-agent.
2. Sub-Agent ($A_S$):
   • Role: A specialized "attribute expert" that analyzes the cropped lesion view.
   • Output: Predicts four clinically relevant structured attributes:
     • Echogenicity pattern
     • Calcification
     • Boundary type
     • Edge (margin) morphology
   • Function: Provides structured, auditable evidence to the Main Agent.

B. Training Strategy: Decoupled Progressive Training

To address training instability and error propagation, the authors introduce a three-stage training pipeline:

1. Stage 1: RL for Sub-Agent (Attribute Specialist)

   • The Sub-Agent is trained using Reinforcement Learning (RL) with a reward function combining attribute accuracy and format compliance.
   • Goal: Create a robust attribute recognition module that produces readable reasoning traces.

2. Stage 2: Oracle-Guided Curriculum RL for Main Agent

   • Problem: Training the Main Agent with predicted (noisy) attributes from the Sub-Agent leads to non-stationary observation distributions.
   • Solution: The Main Agent is trained using Oracle Attributes (Ground Truth attributes) instead of predicted ones.
   • Goal: This decouples the learning of high-level reasoning from early-stage perception noise, allowing the Main Agent to learn stable, attribute-based diagnostic reasoning. The reward focuses solely on diagnostic correctness (Malignancy/BI-RADS).

3. Stage 3: Corrective Trajectory Self-Distillation

   • Problem: The Stage 2 policy relies on oracle data, which is unavailable at test time. Directly using the Stage 2 policy with noisy predictions degrades performance.
   • Solution:
     • Sample trajectories from the Stage 2 policy.
     • Refinement: Replace predicted bounding boxes with Ground Truth boxes (spatial alignment) and rewrite rationales for incorrect diagnoses using the Ground Truth label.
     • Distillation: Use these refined, high-quality trajectories ($\tau^*$) to perform Supervised Fine-Tuning (SFT) on the Main Agent.
   • Goal: Distill the stable reasoning patterns learned in Stage 2 into a deployable end-to-end policy that can handle predicted attributes at test time.

3. Key Contributions

1. Hierarchical Multi-Agent Framework: The first agent-based diagnostic framework for ultrasound that explicitly separates global localization and evidence integration from fine-grained attribute perception, creating an auditable ROI $\to$ Attributes $\to$ Diagnosis evidence chain.
2. Oracle-Guided Curriculum RL: A novel training strategy that uses Ground Truth attributes during the Main Agent's training phase to mitigate non-stationarity and error propagation, ensuring the model learns robust reasoning logic before being exposed to prediction noise.
3. Corrective Trajectory Self-Distillation: A pipeline that refines RL-explored trajectories by injecting spatial and logical supervision, converting sparse/noisy RL signals into dense supervised data for SFT, resulting in a robust deployable policy.

4. Experimental Results

The method was evaluated on three in-domain datasets (BUSBRA, BUSI, BUDIAT) and one out-of-domain (OOD) dataset (BrEaST).

• Performance: UltrasoundAgents achieved state-of-the-art performance, outperforming strong baselines including Zero-shot VLMs, CoT-SFT (Chain-of-Thought with SFT), and single-agent "Think-with-Image" approaches.
  • Overall Metrics: Achieved an AUC of 0.741, Accuracy of 0.813, and Cohen's $\kappa$ of 0.224 on the pooled dataset.
  • OOD Generalization: Showed significant improvement over baselines on the BrEaST dataset (AUC 0.685 vs. 0.586 for CoT-SFT), attributed to the ROI-focused crop-and-zoom workflow reducing background sensitivity.
• Ablation Studies:
  • Removing Oracle Training caused a severe drop in performance (AUC dropped from 0.741 to 0.535), confirming the necessity of decoupling reasoning from perception noise.
  • Removing Self-Distillation significantly reduced localization accuracy (IoU dropped from 0.610 to 0.299), proving the effectiveness of the trajectory refinement step.
• Attribute Recognition: The "Crop-and-Zoom" approach yielded consistently higher F1 scores for boundary, edge, and echo attributes compared to whole-image inputs, demonstrating improved resolution for fine-grained details.

5. Significance

• Clinical Alignment: The framework mirrors the actual clinical reading workflow, making the AI's decision process more intuitive and acceptable to clinicians.
• Auditability: By generating structured intermediate evidence (attributes) and explicit reasoning traces, the system enhances traceability, allowing clinicians to audit why a specific BI-RADS category was assigned.
• Robustness: The proposed training strategy solves the critical challenge of training hierarchical agents where errors in early stages (localization) typically cascade and destroy downstream performance.
• Future Impact: This work provides a blueprint for developing interpretable, multi-step medical AI systems that can be deployed in real-world clinical settings where explainability and reliability are paramount.