EchoAgent: Towards Reliable Echocardiography Interpretation with "Eyes","Hands" and "Minds"

1. Problem Statement

Echocardiography (Echo) is the cornerstone of cardiac diagnosis, but reliable interpretation requires a complex orchestration of three distinct capabilities:

• "Eyes" (Visual Observation): Recognizing cardiac views and observing anatomical structures.
• "Hands" (Manual Operation): Precisely segmenting structures and performing quantitative measurements (e.g., volume, ejection fraction).
• "Minds" (Expert Reasoning): Integrating clinical guidelines, multimodal evidence, and logical reasoning to reach a diagnosis.

Current Limitations:

• Task-Specific Deep Learning (DL) Models: Excel at "eyes" and "hands" (e.g., segmentation, view classification) but lack the "mind" to orchestrate these tasks logically or reason about clinical significance.
• Multimodal Large Language Models (MLLMs): Possess "eyes" and "minds" (visual understanding and reasoning) but lack the specialized "hands" for precise quantitative analysis and domain-specific expertise. They often produce ungrounded, hallucinated, or clinically unreliable results.

There is a critical gap in end-to-end agentic systems that can simultaneously learn, observe, operate, and reason like a human cardiac sonographer.

2. Methodology: EchoAgent Framework

EchoAgent is an agentic system designed to emulate the perceptual-cognitive-motor process of a sonographer. It consists of three core components working in a coordinated workflow:

A. Expertise-Driven Cognition Engine (The "Mind")

• Goal: To construct a domain-specific knowledge base that mimics expert clinical reasoning.
• Mechanism:
  • Ingests heterogeneous textual data from medical libraries (UMLS) and authoritative guidelines (AHA, ASE, EACVI).
  • Uses a Medical Concept Encoder to decompose documents into semantic knowledge primitives.
  • Organizes these primitives into an Anatomy-Indexed Knowledge Repository covering 14 cardiac structures and 48 views.
  • Employs Retrieval-Augmented Generation (RAG) to dynamically retrieve relevant clinical guidelines based on specific anatomical targets or diagnostic queries.

B. Hierarchical Collaboration Toolkit (The "Eyes" and "Hands")

This toolkit provides the operational capabilities through three layers:

1. Perceptual Layer: Uses a specialized Foundation Model (EchoPrime) to parse video streams, identify cardiac views (e.g., Apical 4-chamber), and establish spatial context.
2. Operational Layer: Utilizes customized segmentation models (based on USFM) to automatically delineate key structures (LV, RA, Aorta, etc.).
3. Functional Layer: Employs fine-tuned models to calculate critical clinical parameters (Ejection Fraction, chamber dimensions, pressures) and generate quantitative evidence.

C. Orchestrated Reasoning Hub (The Coordinator)

• Goal: To dynamically integrate the "Mind," "Eyes," and "Hands" into a traceable diagnostic workflow.
• Workflow:
  1. Task Allocation: Retrieves relevant knowledge from the EDC engine and decomposes the diagnostic query into actionable steps mapped to specific tools in the HC toolkit.
  2. Dynamic Reasoning Graph: Constructs a multimodal graph where nodes represent clinical entities (evidence, hypotheses) and edges represent relationships (generation, support, derivation).
  3. Adaptive Inference: Evaluates confidence scores. If uncertainty is high, the hub triggers sub-goals (e.g., requesting a re-measurement from a different view) to refine evidence before drawing a conclusion. This ensures a self-reflective, verifiable reasoning trajectory.

3. Key Contributions

1. Novel Paradigm: Proposes the first specialized agentic system for Echo that unifies "eyes-hands-minds" into a single, coordinated framework, moving beyond isolated task-specific models or passive MLLMs.
2. Architecture Design:
   • An EDC Engine for building a structured, guideline-based knowledge repository.
   • A Hierarchical Collaboration Toolkit enabling automated video parsing, segmentation, and quantitative measurement.
   • An Orchestrated Reasoning Hub that manages a closed-loop, adaptive reasoning process with confidence evaluation.
3. Comprehensive Evaluation: Validated on CAMUS (EF grading) and MIMIC-EchoQA (multi-structure diagnostic QA) datasets, covering 48 views and 14 anatomical regions.

4. Experimental Results

The system was evaluated against task-specific DL models, general-purpose MLLMs, and enhanced MLLMs (with tools added).

• Single-Structure Task (EF Grading on CAMUS):

  • EchoAgent achieved an overall accuracy of 80.00% and a G-mean of 83.87%.
  • It significantly outperformed task-specific models (which lack reasoning) and general MLLMs (which lack quantitative skills).
  • Compared to an enhanced baseline (GPT-5 with tools), EchoAgent showed superior performance, particularly in "Mildly reduced" and "Considerably reduced" EF categories, demonstrating the necessity of the integrated "mind" alongside "hands."
  • AUROC: Achieved 98.43% for detecting systolic dysfunction, indicating high clinical reliability.

• Multi-Structure Task (MIMIC-EchoQA):

  • EchoAgent achieved an average accuracy of 79.42% across 7 anatomical categories.
  • It surpassed the best-performing MLLM (GPT-5) by 49.36% and the best specialized MLLM by 31.45%.
  • Crucially, EchoAgent maintained balanced performance across all structures (e.g., Ventricles, Valves), whereas other models showed severe inconsistencies (e.g., dropping to <30% on specific valves).

• Ablation Studies:

  • Adding the EDC Engine ("Mind") improved accuracy by ~15%.
  • Adding the HC Toolkit ("Hands") improved accuracy by ~37%.
  • The full synergy of Eyes-Hands-Minds yielded the optimal results, proving that coordination is more than the sum of its parts.

5. Significance and Impact

• Clinical Reliability: EchoAgent addresses the "hallucination" and "ungrounded reasoning" issues of current MLLMs by enforcing a workflow grounded in quantitative evidence and clinical guidelines.
• Traceability: The system provides a step-by-step, evidence-backed reasoning chain (visualized in Figure 6), making it suitable for clinical audit and trust.
• Automation Potential: It demonstrates the feasibility of a fully autonomous system that can mimic the entire workflow of a cardiac sonographer, from video ingestion to diagnostic conclusion, potentially reducing the workload on clinicians and improving diagnostic consistency.
• Future Direction: Sets a new standard for medical AI, shifting from isolated task automation to holistic, agentic clinical reasoning.