Tell2Adapt: A Unified Framework for Source Free Unsupervised Domain Adaptation via Vision Foundation Model

Tell2Adapt is a unified Source-Free Unsupervised Domain Adaptation framework that leverages Vision Foundation Models with Context-Aware Prompts Regularization and Visual Plausibility Refinement to achieve state-of-the-art performance across 10 domain shifts and 22 anatomical targets in medical image segmentation.

The Gist

Imagine you are a master chef who has spent years perfecting a recipe for Spicy Tofu using ingredients from a specific farm in Japan. You know exactly how that tofu tastes, how it looks, and how to cook it.

Now, you are asked to cook the same dish for a new restaurant in Brazil, but there's a catch: You cannot take your original Japanese ingredients with you. You only have the new Brazilian tofu, which looks slightly different, has a different texture, and comes from a different farm. If you try to cook using your old Japanese instincts alone, the dish might taste terrible.

This is the problem doctors face with Medical AI. They train AI models on data from one hospital (the "Japanese farm"), but when they try to use that AI at a different hospital with different machines (the "Brazilian farm"), the AI gets confused and makes mistakes. Usually, to fix this, they would need to bring the old data along, but patient privacy laws say, "No, you can't take that data." This is called Source-Free Unsupervised Domain Adaptation (SFUDA). It's like trying to cook the new dish without the old ingredients and without the old recipe book.

Enter Tell2Adapt, a new framework that solves this problem using a "Super-Chef" and a "Quality Inspector."

The Three Magic Ingredients of Tell2Adapt

1. The "Super-Chef" (The Vision Foundation Model)

Think of a Vision Foundation Model (VFM) like a super-intelligent, all-knowing chef who has tasted every dish in the world. This chef doesn't need to see the specific ingredients you have; they just need a clear description of what you want to cook.

In the past, other methods tried to ask the Super-Chef for help by showing them the new Brazilian tofu and asking, "What does this look like?" But because the tofu looked weird, the Super-Chef got confused and gave bad advice.

Tell2Adapt changes the game. Instead of showing the tofu, it asks the Super-Chef using words. It says, "Hey, please find the Liver in this Abdominal CT scan."

2. The "Translator" (Context-Aware Prompts Regularization - CAPR)

Here's the problem: Doctors are busy. Sometimes they type messy instructions like: "liver in abd ct" or "spleen... wait, no, liver... CT scan." If you give these messy notes to the Super-Chef, they might get confused.

CAPR is like a super-efficient secretary. Before the doctor's messy note reaches the Super-Chef, the secretary reads it, fixes the typos, figures out the context (e.g., "Oh, they are looking at an Abdominal CT"), and rewrites it into a perfect, professional sentence: "Liver in Abdominal CT."

This ensures the Super-Chef always gets a crystal-clear instruction, no matter how messy the original request was. The Super-Chef then draws a perfect map (a "pseudo-label") of where the liver should be.

3. The "Quality Inspector" (Visual Plausibility Refinement - VPR)

Once the Super-Chef draws the map, a small, lightweight AI model (the "Student Chef") learns from it to do the job itself. But sometimes, even with a good map, the Student Chef might get a little carried away and draw a liver that looks like a potato or floats in the air.

VPR is the Quality Inspector. It knows the "laws of anatomy." It checks the Student Chef's work and asks:

• "Does this shape look like a real liver?"
• "Is the color and texture consistent with what a liver usually looks like in a CT scan?"

If the Student Chef drew a liver that looks like a potato, the Inspector says, "Nope, that's fake. Throw it out." This removes mistakes and ensures the final result is medically safe and realistic.

Why This is a Big Deal

Before Tell2Adapt, trying to adapt medical AI to new hospitals was like trying to build a house without blueprints, using only guesswork. It worked for small, simple houses (low-gap shifts) but failed for skyscrapers (complex, different medical machines).

Tell2Adapt is different because:

1. It's Universal: It works on brains, hearts, livers, and polyps. It's not just for one specific organ.
2. It's Privacy-Safe: It doesn't need the old data. It just needs the new images and the "Super-Chef's" general knowledge.
3. It's Reliable: By using the "Translator" to clean up instructions and the "Inspector" to check the work, it produces results that are almost as good as if the AI had been trained on the new data from scratch.

The Bottom Line

Tell2Adapt is a smart system that lets medical AI learn new skills on new machines without needing old data. It uses a "Super-Chef" to guide the learning, a "Translator" to make sure the instructions are clear, and a "Quality Inspector" to ensure the final result is safe and accurate. This means doctors can finally use AI tools in more hospitals, saving time and lives, without worrying about privacy or broken models.

Technical Summary

1. Problem Statement

Source-Free Unsupervised Domain Adaptation (SFUDA) is a critical paradigm for deploying deep learning models in clinical settings where access to source data is prohibited due to privacy regulations, yet models must adapt to unseen target domains (e.g., different scanners, modalities, or hospitals).

Current SFUDA methods face two major limitations:

1. Limited Generalization: Most existing approaches are designed for specific anatomical targets or low-gap domain shifts. They lack the flexibility to serve as a unified framework for diverse modalities (CT, MRI, Ultrasound, Endoscopy) and multiple anatomical structures.
2. Error Propagation in VFM Integration: While Vision Foundation Models (VFMs) like SAM or MedSAM offer powerful generalization, previous attempts to use them for SFUDA relied on spatial prompts generated by the source model's predictions. In high-gap domain shifts, these source predictions are often noisy, leading to erroneous prompts that cause the VFM to generate inaccurate pseudo-labels, creating a "failure cascade."

2. Methodology: Tell2Adapt

Tell2Adapt is a unified framework that leverages the generalizable knowledge of a Vision Foundation Model (VFM), specifically BiomedParse (a text-prompted VFM), to adapt a lightweight source model to a target domain without source data. The framework consists of three core components:

A. Context-Aware Prompts Regularization (CAPR)

To overcome the reliance on noisy spatial prompts, Tell2Adapt uses text prompts to guide the VFM. However, real-world clinical prompts can be ambiguous, noisy, or unstructured.

• Mechanism: CAPR utilizes a Large Language Model (LLM, specifically Qwen3-VL-8B-Instruct) to act as a semantic normalizer.
• Process:
  1. Global Context Inference: The LLM analyzes a batch of input prompts to infer shared context (e.g., imaging modality like "CT" and anatomical region like "Abdomen").
  2. Canonicalization: It corrects typos, resolves ambiguities, and enriches sub-prompts with the inferred context.
  3. Output: Prompts are reformatted into a strict canonical structure: [Target] in [Anatomical Site] [Modality] (e.g., "Liver in abdomen CT").
• Benefit: This ensures the VFM receives high-fidelity, standardized instructions, decoupling prompt quality from the source model's unreliable predictions.

B. VFM-Guided Knowledge Distillation

Once high-quality pseudo-labels are generated by the VFM using the regularized prompts, this knowledge is distilled into a lightweight student model.

• Pseudo-Label Generation: BiomedParse generates segmentation masks for the unlabeled target images based on the canonical text prompts.
• Preprocessing: Target images undergo histogram equalization (HE) to mitigate low-level intensity distribution shifts, forcing the model to learn invariant anatomical features.
• Distillation: The source model (e.g., nnUNet) is trained on the target images and the VFM-generated pseudo-labels using Dice and Cross-Entropy losses. This compresses the VFM's generalist knowledge into a compact, deployable specialist model.

C. Visual Plausibility Refinement (VPR)

To ensure clinical reliability, the framework includes a post-processing step to filter out anatomically implausible predictions.

• Mechanism: VPR leverages pre-computed statistical priors (Beta distributions) derived from the VFM regarding the visual properties (pixel probability, intensity in R/G/B channels) of specific anatomical categories.
• Process:
  1. The adapted model generates an initial prediction.
  2. Each disconnected component of the prediction is scored based on how well its visual features align with the statistical priors of the target category.
  3. Components with low plausibility scores (indicating noise or false positives) are removed.
• Benefit: This "re-grounding" step effectively removes artifacts and ensures the final segmentation adheres to known anatomical constraints.

3. Key Contributions

1. Unified SFUDA Framework: Tell2Adapt is the first framework to achieve state-of-the-art (SOTA) performance across 10 domain adaptation directions and 22 distinct anatomical targets (brain, cardiac, polyp, abdominal), demonstrating unprecedented generalization.
2. Context-Aware Prompts Regularization (CAPR): A novel module that transforms noisy, unstructured user inputs into canonical instructions, solving the semantic drift problem and enabling robust VFM guidance without source data.
3. Visual Plausibility Refinement (VPR): A mechanism that enforces anatomical consistency by filtering predictions based on low-level visual statistical priors, significantly reducing false positives in clinical scenarios.
4. Efficiency: The framework distills the massive VFM (371M parameters) into a lightweight student model (31M parameters), reducing peak VRAM usage from 26.1 GB to 4.8 GB, making it practical for clinical deployment.

4. Experimental Results

The authors conducted extensive evaluations on datasets including AMOS (Abdomen), BraTS (Brain), CAMUS/ACDC (Cardiac), and Kvasir/CVCDB (Polyps).

• Abdominal Targets (AMOS): In MR→CT adaptation, Tell2Adapt achieved a mean Dice score of 88.2%, nearly matching the fully supervised upper bound (88.4%) and significantly outperforming all baselines (e.g., Baseline: 47.4%).
• Brain Targets (BraTS): The framework showed robust performance across four modalities (T1n, T1c, T2w, T2f). For example, in T1c→T2f, it achieved 86.0% Dice for the Surrounding Non-Enhancing FLAIR Hyperintensity (SNFH) region, surpassing competitors by a large margin.
• Cardiac Targets (ACDC/CAMUS): In the extremely challenging cross-modality shift from MRI to Ultrasound (US), TellAdapt achieved 94.6% Dice for the Left Ventricle, whereas baseline methods failed completely (near 0% Dice).
• Polyp Targets: Showed consistent gains, achieving 89.1% Dice in Kvasir→CVCDB, outperforming the second-best method by over 11%.
• Ablation Studies:
  • Removing CAPR caused performance to collapse (e.g., dropping to ~48% Dice in abdominal tasks with chaotic prompts), proving its necessity for handling noisy inputs.
  • Removing VPR resulted in a ~3-4% drop in Dice and increased Average Surface Distance (ASD), confirming its role in refining boundary accuracy.

5. Significance

Tell2Adapt represents a paradigm shift in medical image analysis by:

• Breaking the Source Data Dependency: It enables robust adaptation in privacy-sensitive environments where source data is unavailable.
• Solving the "High-Gap" Problem: Unlike previous methods that fail under severe domain shifts (e.g., MRI to Ultrasound), Tell2Adapt leverages the VFM's broad knowledge to bridge these gaps effectively.
• Establishing a New Standard: It moves SFUDA from specialized, single-task solutions to a unified, multi-modal, and multi-target framework, setting a new benchmark for clinical deployability and generalization.
• Clinical Reliability: By incorporating VPR, the framework addresses the critical need for anatomical plausibility, reducing the risk of false positives in real-world deployment.

The code and models are made available on GitHub to ensure reproducibility.