FetalAgents: A Multi-Agent System for Fetal Ultrasound Image and Video Analysis

FetalAgents is a novel multi-agent system that dynamically orchestrates specialized vision experts to deliver robust, end-to-end fetal ultrasound analysis and structured clinical reporting across multiple tasks, outperforming existing specialized models and multimodal large language models.

The Gist

Imagine you are trying to read a very complex, blurry map of a tiny, moving world inside a mother's belly. This is what a fetal ultrasound is. For decades, only highly trained experts (sonographers) could look at these grainy black-and-white images, figure out what they were seeing, measure the baby's parts, and write a report. It's a job that requires years of practice, and there aren't enough experts to go around.

Recently, computers (AI) got really good at doing one specific thing: maybe just finding the baby's head, or just measuring the heart. But they were like specialized robots that could only do one task. If you wanted a full report, a human doctor still had to run ten different robots, collect their answers, and write the report themselves.

Enter FetalAgents. Think of this not as a single robot, but as a highly efficient hospital team working together in real-time.

The Team Structure: A "Brain" and a "Crew"

The paper describes FetalAgents as a Multi-Agent System. Here is how it works using a simple analogy:

1. The Coordinator (The "Project Manager")
Imagine a smart project manager sitting at a desk. When a doctor asks, "What do we see here?" or uploads a video, this manager doesn't try to do the measuring themselves. Instead, they look at the request, figure out what kind of image it is (is it a brain? a tummy?), and immediately calls the right person for the job. They are the conductor of the orchestra.

2. The Experts (The "Specialists")
These are the "workers" the manager calls.

• The Plane Expert: "Is this a picture of the baby's head or the stomach?"
• The Measurer: "Let me draw a line around the head to get the circumference."
• The Biometry Expert: "Now, let's calculate the age based on these measurements."
• The Video Expert: "I'm watching the whole video clip to find the clearest moments."

Crucially, these experts don't just guess. They are built on top of the best existing AI models. If one model is shaky, the team combines (or "ensembles") the opinions of three or four different models to make sure the answer is rock-solid. They talk to each other using strict, clear data formats (like filling out a standardized form) so they don't get confused or "hallucinate" (make things up).

3. The Summarizer (The "Reporter")
Once the specialists have done their work, the Summarizer steps in. This agent takes all the raw numbers and facts, checks them against medical rules (like "Does this head size make sense for a 20-week-old baby?"), and writes a final, easy-to-read clinical report. It's like a journalist who takes interview notes from five different sources and writes a perfect news article.

The Magic Trick: From Static Photos to Live Movies

Most AI tools today are like photographers who can only analyze a single, frozen photo. But real ultrasound is a movie. The baby moves, the probe moves, and the doctor needs to scan through the whole video to find the best angles.

FetalAgents is unique because it acts like a smart video editor.

• It watches the entire continuous video stream.
• It automatically skips the blurry or useless parts.
• It picks out the "Key Frames" (the best, clearest shots of the brain, heart, and tummy).
• It analyzes those specific frames, measures the baby, and then writes a summary of the entire video session.

Why Does This Matter?

The paper tested this system against:

1. Single-specialist AI: The "one-trick ponies."
2. General AI (LLMs): The smart chatbots that know a lot about everything but aren't great at medical math.
3. Human Experts: The gold standard.

The Result: FetalAgents beat the single-specialist AI and the general chatbots. It was more accurate, more consistent, and better at handling the messy reality of real-world hospital videos.

The Bottom Line

FetalAgents is like upgrading from a toolbox full of separate, single-purpose tools (a hammer, a screwdriver, a wrench) to a smart, self-assembling robot crew.

• For Doctors: It handles the boring, repetitive math and video editing, letting them focus on patient care.
• For Patients: It means faster, more accurate diagnoses, even in places where expert sonographers are scarce.
• For the Future: It proves that AI doesn't just need to be a "calculator"; it can be a collaborator that understands the whole workflow, from the first scan to the final report.

In short, FetalAgents doesn't just "look" at the baby; it thinks like a sonographer, works like a team, and reports like a doctor.

Technical Summary

Here is a detailed technical summary of the paper "FetalAgents: A Multi-Agent System for Fetal Ultrasound Image and Video Analysis."

1. Problem Statement

Fetal ultrasound (US) is the primary modality for prenatal screening, but its interpretation relies heavily on the expertise of sonographers, leading to disparities in care quality due to a shortage of experienced clinicians. While deep learning has advanced automated analysis, existing solutions suffer from two critical limitations:

• Task Fragmentation: Current models are narrowly specialized (e.g., a model only for segmentation or only for classification) and cannot handle end-to-end clinical workflows.
• Lack of Versatility: Most tools process static images in isolation, failing to support temporal video stream summarization or the synthesis of structured clinical reports that integrate multiple anatomical planes and patient metadata.
• Workflow Disconnect: Clinicians must manually coordinate between different tools and translate raw predictions into documentation, a process that is labor-intensive and prone to error.

2. Methodology: The FetalAgents Framework

The authors propose FetalAgents, the first multi-agent system designed to bridge the gap between isolated point-prediction models and holistic clinical workflows. Built on the AutoGen framework, the system employs a lightweight, agentic coordination architecture consisting of three distinct agent types:

A. Agent Roles

1. Coordinator Agent (The Orchestrator):
   • Based on GPT-5-mini, this agent acts as the central brain.
   • It interprets user queries and input data (images or video streams).
   • It classifies the task type (e.g., diagnosis, measurement, reporting) and identifies the anatomical plane.
   • It dynamically dispatches structured prompts to relevant Expert Agents.
2. Expert Agents (The Specialists):
   • These are modular agents dedicated to specific clinical tasks: Plane Classification, Anatomical Segmentation, and Fetal Biometry.
   • Each expert wraps a suite of state-of-the-art vision models (e.g., FetalCLIP, USFM, nnU-Net, ViT, ResNet) as callable tools.
   • They employ deterministic fusion rules to ensemble outputs from multiple models, ensuring robustness.
   • Crucially, they communicate results in structured JSON formats to prevent hallucinations from intermediate steps.
3. Summarizer Agent (The Reporter):
   • Aggregates structured outputs from experts and tool-decision notes.
   • Cross-references biometric findings with patient metadata (e.g., Last Menstrual Period - LMP) and WHO growth charts.
   • Acts as a clinical safeguard: If biometric measurements deviate significantly from growth norms, it triggers a reflection step to re-evaluate outlier predictions before generating the final report.

B. Core Capabilities

• Image Analysis: Performs plane classification, anatomical segmentation (head, abdomen, stomach), and biometry (Head Circumference, Abdominal Circumference, Angle of Progression).
• Video Summarization: A novel extension where the system processes continuous US video streams. It uses a fine-tuned FetalCLIP-based encoder to automatically identify diagnostic keyframes across multiple anatomical planes, analyzes them frame-by-frame, and synthesizes a holistic, video-level clinical report.

3. Key Contributions

1. First Multi-Agent System for Fetal US: Introduces a framework that dynamically orchestrates specialized vision experts to emulate the holistic reasoning of human sonographers.
2. End-to-End Video Workflow: Moves beyond static image analysis to automate the extraction of keyframes and the generation of structured reports from continuous video streams, a significant leap in clinical utility.
3. Agentic Coordination & Ensembling: Demonstrates that a lightweight coordination layer can effectively combine diverse foundation models (FetalCLIP, USFM, SAM-based models) to outperform any single monolithic model.
4. Clinical Consistency Safeguards: Integrates WHO growth charts and patient metadata (LMP) directly into the generation pipeline to validate biometric plausibility and ensure report accuracy.

4. Experimental Results

The system was evaluated on eight clinical tasks across multiple external, multi-center datasets (including African data and various public benchmarks), comparing against specialized vision models, foundation models, and Multimodal Large Language Models (MLLMs).

• Plane Classification: FetalAgents achieved 92.7% accuracy on standard planes (vs. 91.4% for FetalCLIP) and 90.9% accuracy on brain sub-planes, significantly outperforming general MLLMs like GPT-5-mini (60.5% accuracy).
• Anatomical Segmentation:
  • Abdomen: Achieved a Dice Similarity Coefficient (DSC) of 0.937 and Hausdorff Distance (HD95) of 2.32, surpassing the runner-up (FetalCLIP) by 3.5% in DSC.
  • Stomach: Achieved a DSC of 0.859, outperforming nnU-Net.
• Fetal Biometry:
  • AC Measurement: Reduced Mean Absolute Error (MAE) from 9.94 mm (FetalCLIP) to 5.53 mm.
  • GA Prediction: Achieved the highest validity rate (84.93%) and lowest MAE (1.23 weeks), outperforming specialized regression models.
• Qualitative Performance: In case studies, the system successfully generated structured reports that matched human expert annotations, correctly identified keyframes in video streams, and flagged inconsistencies between ultrasound-derived GA and LMP-derived GA.

5. Significance and Impact

• Workflow Automation: FetalAgents addresses the "last mile" problem in medical AI by automating the entire workflow from raw video input to structured clinical documentation, reducing the burden on sonographers.
• Robustness: By using an ensemble approach coordinated by an agent, the system mitigates the weaknesses of individual models, providing more reliable results in diverse, real-world clinical settings (including under-resourced regions).
• Auditability: The system provides a transparent, step-by-step reasoning process (via the Coordinator and Summarizer), making the AI's decision-making auditable and trustworthy for clinical adoption.
• Future Direction: This work sets a precedent for "Agentic Medical AI," suggesting that future systems should move beyond single-task predictors toward multi-agent collaborations that handle complex, temporal, and multi-modal clinical data.