Adapting Medical Vision Foundation Models for Volumetric Medical Image Segmentation via Active Learning and Selective Semi-supervised Fine-tuning

This paper proposes the Active Selective Semi-supervised Fine-tuning (ASSFT) framework, which enhances the adaptation of medical vision foundation models for volumetric segmentation by combining an active learning strategy that selects informative samples based on knowledge divergence and anatomical difficulty with a semi-supervised approach that leverages reliable unlabeled data to maximize performance under limited annotation budgets.

The Gist

Imagine you have a brilliant medical student who has spent years studying millions of generic anatomy textbooks (this is the Medical Vision Foundation Model, or Med-VFM). They know the human body inside out, but they've never seen a specific type of MRI machine or a unique hospital's patient data before.

Now, you want this student to start working in a new hospital (the Target Domain) to help doctors segment organs (like drawing outlines around the liver or kidneys) on 3D scans. The problem? The new hospital's scans look slightly different, and the student hasn't been trained on them yet. If you just let them guess, they'll make mistakes. If you ask them to study every single new scan and have a human expert label them, it would take forever and cost a fortune.

This paper introduces a smart, efficient way to train this student: Active Selective Semi-supervised Fine-tuning (ASSFT). Think of it as a "Super Tutor" system that helps the student learn the new hospital's specific style using the fewest possible examples.

Here is how the system works, broken down into simple steps:

1. The "Super Tutor" Strategy (Active Learning)

Instead of asking the student to study random scans, the system acts like a smart tutor who knows exactly which examples will teach the student the most.

The system uses two special "glasses" to pick the best scans to show the student:

• Glasses #1: The "Knowledge Gap" Lens (DKD)
  Imagine the student has a mental map of the body. This lens looks for scans where the student's map is completely wrong or missing pieces. It asks: "Does this scan show something the student has never seen before?" If the answer is yes, it's a high-priority study item. It also makes sure the student doesn't just study the same type of weird liver twice; it ensures they see a variety of new things.
• Glasses #2: The "Tricky Anatomy" Lens (ASD)
  Sometimes, a scan might be confusing not because it's new, but because the organ is weirdly shaped or hard to see. This lens looks specifically at the organs (the foreground) and ignores the empty space (the background). It asks: "Is this organ hard to outline?" If the student is struggling to guess where the kidney ends and the muscle begins, this lens flags that scan as a top priority for study.

The Result: The system picks only the most confusing and unique scans, asks a human expert to label them, and then teaches the student. This saves a massive amount of time because the student learns from the "hard stuff" first.

2. The "Confident Guessing" Strategy (Selective Semi-supervised Learning)

Once the student has learned from the expert-labeled examples, there are still thousands of unlabeled scans sitting in the pile. The system doesn't ignore them. Instead, it lets the student try to label them on their own, but with a safety net.

• The Safety Net: The system only lets the student "self-study" scans where the student is very confident and where the scan looks very similar to the ones the expert already labeled.
• The Filter: If the student is unsure or the scan looks totally different from what they've learned, the system says, "No, don't guess on this one yet." This prevents the student from learning bad habits (wrong labels) from their own mistakes.

3. The Loop

The process repeats in a cycle:

1. Pick the best new examples using the two lenses (Knowledge Gap + Tricky Anatomy).
2. Get them labeled by a human.
3. Let the student study these new labels plus the "safe" unlabeled ones they guessed correctly.
4. Repeat until the student is an expert on the new hospital's data.

Why is this a big deal?

The paper tested this on five different medical datasets (different body parts, different types of scans like CT and MRI). They found that:

• It's faster: The system reached expert-level performance using only a tiny fraction of the labeled data that traditional methods need.
• It's smarter: It consistently beat other methods that just picked random scans or only looked at "uncertainty."
• It works without the old data: Usually, to adapt a model, you need to see the original training data. This system works even if that original data is locked away for privacy reasons.

In short: This paper gives medical AI a way to learn a new job quickly by studying only the most interesting and difficult examples, while carefully ignoring the easy stuff and the confusing guesses. It turns a "one-size-fits-all" AI into a specialized expert with very little human help.

Technical Summary

Technical Summary: Active Selective Semi-supervised Fine-tuning for Medical Vision Foundation Models

1. Problem Statement

Medical Vision Foundation Models (Med-VFMs), pre-trained on large-scale unlabeled medical datasets via self-supervised learning, have demonstrated strong potential for medical image analysis. However, their performance in downstream tasks, particularly volumetric medical image segmentation, remains limited when applied to new target domains.

Current adaptation strategies face three primary limitations:

1. Inefficient Sample Selection: Existing Active Learning (AL) and Active Domain Adaptation (ADA) methods often rely on random sampling or simple uncertainty/diversity metrics. These approaches fail to explicitly leverage the pre-trained knowledge of Med-VFMs to identify samples containing "unlearned" target-domain patterns. Furthermore, image-level metrics often bias selection toward background uncertainty, neglecting informative foreground anatomical structures.
2. Source Data Dependency: Many domain adaptation methods require access to source-domain data to guide adaptation. In practice, pre-training data for Med-VFMs is often unavailable due to privacy constraints, rendering these methods inapplicable.
3. Noisy Semi-supervised Training: While semi-supervised learning (SSL) can utilize abundant unlabeled target data, naively using all pseudo-labeled samples introduces noise, especially in early adaptation rounds when the model is not yet reliable. This can degrade performance or cause the model to overfit to noisy pseudo-labels rather than learning from high-quality labeled data.

The central challenge is to adapt Med-VFMs to target domains efficiently under a limited annotation budget, without source data, while maximizing the utility of both labeled and unlabeled target samples.

2. Methodology: Active Selective Semi-supervised Fine-tuning (ASSFT)

The authors propose ASSFT, a framework that integrates an active learning strategy with a selective semi-supervised fine-tuning mechanism. The framework operates iteratively over $R$ rounds without requiring access to source-domain data.

2.1. Active Test-Time Sample Query Strategy

To select the most informative samples for annotation, the authors introduce a query strategy based on two complementary metrics: Diversified Knowledge Divergence (DKD) and Anatomical Segmentation Difficulty (ASD).

• Diversified Knowledge Divergence (DKD): This metric identifies samples that introduce new knowledge relative to the pre-trained model while ensuring diversity within the target dataset. It comprises two components:

  • Prior and Adaptive Knowledge Divergence (PAKD): Measures the cosine distance between feature embeddings from the initial pre-trained encoder $E^{(0)}$ and the adapted encoder $E^{(i)}$. High PAKD indicates the sample contains domain-specific information not yet captured by the model.
  • Pairwise Dissimilarity (PD): Measures the semantic dissimilarity of a candidate sample relative to previously ranked high-PAKD samples to avoid redundancy and promote intra-domain diversity.
  • DKD Score: Defined as the product of PAKD and PD.

• Anatomical Segmentation Difficulty (ASD): This metric focuses on the difficulty of segmenting foreground anatomical structures rather than the entire image volume.

  • To prevent background dominance, a temperature scaling mechanism $\tau(r)$ is applied to the background class probability, dynamically decreasing from 3 to 1.5 over the adaptation rounds.
  • A binary foreground mask is generated based on the adjusted probabilities.
  • The ASD score is computed as the entropy of class probabilities within the foreground region. High ASD indicates complex anatomical patterns that are challenging for the model.

• Unified Query Criterion: DKD and ASD scores are normalized and transformed via quantile mapping to ensure comparability, then summed to form the final query score $Q(x)$. The top $N_B$ samples are selected for expert annotation.

2.2. Selective Semi-supervised Fine-tuning

To leverage unlabeled data without introducing noise, the framework employs a three-stage process in each round:

1. Supervised Fine-tuning: The model is first updated using the currently available labeled target samples.
2. Reliable Unlabeled Sample Selection: A subset of unlabeled samples is selected for pseudo-labeling based on:
   • Predictive Confidence: The margin between the top two predicted class probabilities in foreground regions.
   • Semantic Distance: The minimum cosine distance between the candidate sample's feature embedding and the embeddings of labeled samples (anchors).
   • Samples with high confidence and small semantic distance are deemed reliable. The number of selected samples ($N_{SU}$) increases with the iteration number ($N_{SU} = N_B \cdot r$).
3. Pseudo-label-based Fine-tuning: Pseudo-labels are generated for the selected reliable samples. These are combined with the labeled set to form an augmented training set for further fine-tuning.

Note: Samples selected for pseudo-labeling are explicitly excluded from the candidate pool for the next active learning round to avoid redundant annotation.

3. Key Contributions

1. ASSFT Framework: A unified framework for adapting Med-VFMs to volumetric segmentation tasks that integrates active learning and selective semi-supervised learning, operating without source-domain data.
2. Active Test-Time Sample Query: A novel strategy utilizing DKD and ASD to select informative samples. DKD captures knowledge novelty and diversity, while ASD prioritizes anatomical complexity, addressing the limitations of standard uncertainty-based methods.
3. Selective Semi-supervised Fine-tuning: A mechanism that selectively incorporates reliable unlabeled samples based on predictive confidence and semantic proximity to labeled data, mitigating the risks of noisy pseudo-labels.
4. Extensive Validation: Comprehensive experiments across five diverse volumetric medical image segmentation tasks (different modalities, anatomical structures, and dataset scales).

4. Experimental Results

The authors evaluated ASSFT on five datasets: AMOS2022-CT, FLARE 2021, Abdomen Atlas, AMOS2022-MRI, and Abdominal MRI.

• Performance: ASSFT consistently outperformed state-of-the-art AL and ADA methods (including Random, Entropy, Core-set, BADGE, SANN, UGTST, and CUP) across all datasets and query budgets.
  • On AMOS2022-CT, with only 5% queried samples, ASSFT achieved a Dice score of 80.51, outperforming the strong baseline UGTST by ~4.7 points and Random selection by ~7.2 points.
  • On AMOS2022-MRI (cross-modality adaptation), ASSFT improved the Dice score from a near-zero-shot baseline of 0.46 to 52.06 with 5% queried samples, a gain of over 51 points.
  • On Abdominal MRI (few-shot setting), ASSFT achieved a Dice of 83.98 with only 3 labeled samples (3-shot), significantly outperforming other methods.
• Efficiency: The method rapidly approaches the fully supervised upper-bound performance (100% labeled data) with a fraction of the annotation cost. For instance, on FLARE 2021, 25% queried samples allowed the model to reach 97.96% of the fully supervised performance.
• Ablation Studies:
  • Removing the semi-supervised component (DKD+ASD only) resulted in lower performance, confirming the value of selective pseudo-labeling.
  • Using only PAKD or PD individually was inferior to the combined DKD metric.
  • The dynamic temperature scaling in ASD was shown to be superior to fixed temperature or no masking.
  • Statistical analysis (Mann-Whitney U test) confirmed that samples selected for pseudo-labeling had significantly higher Dice scores than unselected samples ($p < 0.01$).

5. Significance and Claims

The paper claims that ASSFT provides an annotation-efficient and generalizable solution for deploying Med-VFMs in clinical settings where:

• Source data is unavailable: The method operates in a source-free domain adaptation setting, crucial for privacy-constrained medical data.
• Annotations are scarce: By actively selecting the most informative samples and leveraging reliable unlabeled data, the framework achieves high performance with minimal expert labeling.
• Domain shift is significant: The framework demonstrates robustness across different imaging modalities (CT to MRI) and varying anatomical complexities.

The authors emphasize that their approach addresses the specific limitations of applying foundation models to medical segmentation, particularly the need to balance knowledge novelty, data diversity, and task-specific anatomical difficulty. They conclude that ASSFT facilitates the translation of Med-VFMs into practical clinical workflows by significantly reducing the annotation burden while maintaining high segmentation accuracy.