💻 computer science

UltraSAM3: A Concept-Driven Foundation Model for Universal Ultrasound Image Segmentation

UltraSAM3 is a concept-driven foundation model that leverages text-based target specification and an instruction-guided agent to achieve robust, universal ultrasound image segmentation across diverse organs and lesions, outperforming existing task-specific and visual-prompt-based methods.

Original authors: Bo Xu, Quanhao Zhu, Rui Lin, Boling Zhu, Chenyuan Wang, Hongfei Lin, Feng Xia, Chenhua Ji

Published 2026-08-03

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Bo Xu, Quanhao Zhu, Rui Lin, Boling Zhu, Chenyuan Wang, Hongfei Lin, Feng Xia, Chenhua Ji

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific toy in a giant, messy sandbox. If you just say "find the red car," a helpful robot might scan the whole box and point it out. But what if the sand is swirling, the light is dim, and the red car looks a bit like a red bucket? That's exactly the problem doctors face with ultrasound imaging. Ultrasound is like a magical, real-time X-ray that uses sound waves instead of radiation. It's cheap, safe, and portable, making it a favorite for checking everything from a baby's heartbeat to a sore muscle. However, the pictures it creates are notoriously tricky. They are often grainy (like static on an old TV), have fuzzy edges, and can look very different depending on who is holding the probe.

For years, computer programs designed to "see" these images have been like specialized robots: one robot only knows how to find a baby's head, another only knows how to find a liver, and a third only looks for thyroid nodules. If a doctor wanted to switch tasks, they had to swap robots. More recently, scientists built "foundation models"—super-smart AI brains trained on millions of images that can understand general concepts. But even these giants struggle with ultrasound because the "noise" in the pictures confuses them. They might understand the word "kidney" but fail to find it in a grainy ultrasound because they were trained mostly on clear CT scans or MRIs. The big question was: Can we build a single, smart robot that understands medical concepts and can find any organ or lesion in a noisy ultrasound image just by listening to a simple description?

Enter UltraSAM3, a new invention by a team of researchers that acts like a universal translator for ultrasound images. Think of it as a super-powered detective that doesn't just look at a picture; it listens to a short, specific clue like "find the left ventricle" or "spot the thyroid nodule" and instantly draws a perfect outline around it, even in the messiest, grainiest ultrasound scan.

The Problem with Old Robots

Before UltraSAM3, the best tools for this job were either "task-specific" or "prompt-based."

Task-specific models were like a key that only opens one door. If you trained a model to find breast lumps, it couldn't suddenly find a baby's head. Doctors had to use a different model for every single body part, which is slow and clunky.
Prompt-based models (like the famous SAM family) were a step up. They let users draw a box or a dot on the screen to say, "Look here." But in a real clinic, a doctor doesn't always have time to draw a perfect box. They just want to say, "Show me the carotid artery." If the doctor has to manually point at the blurry image first, the system isn't very helpful.

The UltraSAM3 Solution

The researchers behind UltraSAM3 decided to teach a powerful AI model (called SAM3) to speak "Ultrasound" and "Medical Concepts" at the same time. Instead of just showing it pictures and boxes, they fed it a massive library of 37 different ultrasound datasets covering 13 different body parts (like the heart, liver, fetus, and nerves).

Here is the magic trick: They taught the model using triplets. Imagine a flashcard that has three things on it:

The Ultrasound Image (the grainy picture).
The Mask (the perfect outline of the object).
The Concept (a simple text label like "fetal head" or "kidney").

By seeing millions of these triplets, UltraSAM3 learned to connect the messy, noisy visual patterns of an ultrasound with the clear meaning of medical words. It learned that even though a "thyroid" looks different in every single scan, the word "thyroid" always points to a specific shape in that noise. This allows the model to skip the "draw a box" step. A doctor can just type "segment the thyroid," and the AI knows exactly what to do.

The "Instruction-Guided Agent"

The researchers realized that sometimes doctors might type long, complicated sentences like, "Can you please look at this image and find the suspicious lump in the breast?" While UltraSAM3 is smart, long sentences can sometimes confuse it. To fix this, they built a little helper called an Instruction-Guided Agent.

Think of this agent as a translator or a secretary. When a doctor types a complex request, the agent quickly reads it, figures out the most important part (e.g., "breast lesion"), and rewrites it into a short, punchy command for UltraSAM3. It's like the agent saying, "Okay, I heard you want the breast lump. I'll tell the robot to look for a 'breast lesion'." This makes the whole process smoother and more accurate, especially when the doctor's instructions are wordy or vague.

What They Found

The team tested UltraSAM3 on a huge variety of ultrasound images, including some it had never seen before. The results were impressive:

Better than the competition: On 13 different body parts, UltraSAM3 consistently beat other top models. For example, on thyroid images, it improved the accuracy (measured by a score called Dice) by a massive 0.8297 compared to the next best model, which was barely above zero.
Handling the noise: It worked well even on difficult images with low contrast or heavy grain, proving that teaching it specifically about ultrasound "noise" was the right move.
The Agent helps: When they used the instruction-guided agent to translate complex sentences, the accuracy jumped even higher, improving the average score by 0.161 across all organs. This showed that breaking down complex requests into simple concepts really helps the AI focus.

Why This Matters

The paper suggests that UltraSAM3 is a significant step forward because it moves away from the idea that we need a different robot for every body part. Instead, it offers a universal tool that can handle many different tasks just by listening to text. It suggests that by teaching AI to understand the specific "language" of ultrasound images (the noise, the shadows, the grain), we can make these tools much more useful in real hospitals.

The researchers are careful to note that while the results are strong, this is a new tool that needs to be tested in real-world clinics. They don't claim it's perfect or that it replaces doctors, but rather that it provides a flexible, interactive way to help doctors see what they need to see, faster and more accurately. By turning complex medical questions into simple, actionable commands, UltraSAM3 aims to make ultrasound imaging more accessible and powerful for everyone.

Technical Summary: UltraSAM3

Problem Statement
Ultrasound imaging is a cornerstone of clinical practice due to its portability, cost-effectiveness, and real-time capabilities. However, segmenting ultrasound images remains a significant challenge compared to modalities like CT or MRI. Ultrasound images are characterized by speckle noise, low contrast, acoustic shadows, ambiguous boundaries, and strong operator dependence. Existing segmentation approaches suffer from two primary limitations:

Task-Specific Models: Traditional methods (e.g., U-Net, nnU-Net, Transformer-based variants) are optimized for fixed label spaces and specific imaging distributions. They require separate models for different organs or lesions (e.g., distinct models for fetal heads vs. thyroid nodules), limiting scalability and flexibility in diverse clinical scenarios.
Visual-Prompt Limitations: Recent foundation models like SAM and SAM2 allow for promptable segmentation but rely heavily on visual prompts (points, boxes, masks). In clinical practice, requiring a user to manually localize a target with a box or point assumes the target is already identified, which reduces system autonomy and fails to support natural language requests like "segment the fetal head."

While concept-driven models (e.g., SAM3, MedSAM3) have emerged to support text-based segmentation, they are generally designed for multi-modal or general biomedical imaging. They often struggle to align semantic concepts with the noisy, low-contrast, and artifact-prone visual patterns specific to ultrasound.

Methodology
The authors propose UltraSAM3, a concept-driven foundation model specifically adapted for universal ultrasound image segmentation. The framework consists of two main components:

UltraSAM3 Model Architecture:
- Base: The model is built upon the SAM3 architecture, which advances segmentation from visual prompting to concept-based prompting.
- Training Strategy: Unlike approaches that freeze encoders, UltraSAM3 undergoes full-parameter fine-tuning. This includes updating the image encoder, text encoder, detector, tracker, and segmentation modules.
- Data Formulation: The model is trained on a large-scale corpus comprising 37 public ultrasound datasets covering 13 anatomical categories (lung, muscle, abdomen, ovary, heart, prostate, thyroid, liver, fetus, nerve, breast, carotid artery, and kidney).
- Triplet Learning: Each training sample is formulated as an image–mask–concept triplet $(I, M, q)$ . The textual concept $q$ is derived from the organ, lesion, or anatomical label. This formulation forces the model to learn associations between ultrasound-specific visual patterns and standardized medical concepts without relying on privileged spatial cues like ground-truth bounding boxes.
Instruction-Guided Agent:
- To address the issue where complex natural language instructions degrade performance, the authors introduce an instruction-guided agent acting as a front-end parser.
- Function: Given a complex user query $u$ , an ultrasound image $I$ , and an organ domain $o$ , the agent (implemented using Gemini 3.1 Pro) identifies the target organ, selects the most relevant concept from a candidate pool, and rewrites the instruction into a concise, SAM3-compatible concept prompt $q^*$ .
- Pipeline: The agent decomposes the task into high-level semantic understanding and low-level mask prediction, passing the refined prompt to UltraSAM3 for segmentation.

Key Contributions

First Ultrasound-Specific Concept-Driven Foundation Model: UltraSAM3 is presented as the first model of its kind to adapt SAM3 specifically for ultrasound, enabling text-based target specification without visual prompts.
Instruction-Guided Agent: The development of a mechanism to parse complex clinical queries into robust concept prompts, enhancing the usability of the foundation model in realistic interaction settings.
Comprehensive Evaluation: Extensive experiments across 37 datasets and 13 anatomical categories demonstrate state-of-the-art performance and strong generalization capabilities.

Experimental Results
The paper evaluates UltraSAM3 against representative concept- and text-driven baselines (UniBiomed, BiomedParse, SAM3, Medical SAM3) and visual-prompt-enhanced settings.

Performance on Representative Datasets: On 13 anatomical categories, UltraSAM3 achieved an average IoU of 0.6342 and Dice of 0.7144. This represents an absolute improvement of +0.3088 in IoU and +0.3132 in Dice over the strongest baseline. Significant gains were observed in challenging categories such as thyroid, muscle, prostate, and cardiac structures.
Generalization: On four external datasets not used in training (BrEast, CCAUI, 105US_tumor, KFGNet), UltraSAM3 consistently outperformed all baselines, demonstrating robust transferability to unseen ultrasound domains.
Visual Prompt Robustness: When visual prompts (text + image) were provided, UltraSAM3 still maintained superior performance over other models, indicating that its ultrasound-specific adaptation benefits segmentation even with additional visual guidance.
Agent Effectiveness: The instruction-guided agent improved segmentation robustness, yielding an average Dice gain of +0.161 over directly feeding complex instructions into the model. The agent was particularly effective for breast, liver, thyroid, lung, and nerve images.
Qualitative Analysis: Visual comparisons showed UltraSAM3 producing masks more spatially consistent with ground truth, better capturing anatomical extents, and handling small structures and ambiguous boundaries more effectively than baselines.

Significance and Claims
The authors claim that UltraSAM3 demonstrates that ultrasound-specific concept adaptation is effective for building generalizable and interactive segmentation foundation models. By aligning ultrasound visual patterns with clinically meaningful concepts across diverse organs and acquisition conditions, the model overcomes the limitations of task-specific models and the dependency on visual prompts found in general foundation models. The work suggests that a dedicated domain adaptation strategy is necessary to handle the unique acoustic artifacts and low signal-to-noise ratios of ultrasound, enabling more flexible and autonomous clinical assistance. The code is made available to facilitate further research in this domain.