Technical Summary: CM-TTA for SAM3 in Medical Image Segmentation

Problem Statement

While vision-language models like the Segment Anything Model 3 (SAM3) demonstrate strong zero-shot generalization on natural images, their performance significantly degrades in medical imaging due to substantial domain gaps in imaging principles and styles. Existing Test-Time Adaptation (TTA) methods face two primary limitations in this context:

1. Inadequate Evaluation Metrics: Current vision-language TTA approaches rely heavily on image-level uncertainty minimization (e.g., entropy or variance). These metrics often fail to reflect region-level semantic correctness, which is critical for precise medical segmentation.
2. Instability in Continual Adaptation: Existing methods typically track only instantaneously updated parameters, making them sensitive to the current test batch and prone to instability during long-term, one-pass adaptation where reliable dense supervision (ground truth) is unavailable.

Methodology: CM-TTA

The authors propose Concept Alignment Contrast and Long-Short Prompt Memory for Test-Time Adaptation (CM-TTA). This framework adapts SAM3 to unseen medical domains via prompt tuning, updating only the learnable text prompt token embeddings ($T$) while keeping the vision encoder, text encoder, and detector frozen. The method operates under a one-pass continual TTA protocol, processing each test sample exactly once.

The framework comprises three core components:

1. Concept Alignment Contrast (CAC)

To address the lack of semantic guidance in traditional uncertainty metrics, CAC introduces a text-guided quality score.

• Mechanism: For a test image with multiple augmented views, the method computes the cosine similarity between the textual prompt embedding and the visual features of the predicted foreground and background regions.
• Score Calculation: The CAC score ($C_i$) is defined as the difference between the foreground-text similarity and the background-text similarity: $C_i = \text{sim}(\bar{z}_{i,fg}, t_i) - \text{sim}(\bar{z}_{i,bg}, t_i)$.
• Function: This metric explicitly leverages text-driven semantic consistency to evaluate prediction quality. The augmented view with the highest CAC score is selected as the optimal view ($X^*_i$) for subsequent adaptation steps.

2. Long-Short Prompt Memory (LSPM)

To balance rapid local adaptation with long-term stability, the authors design a dual-memory module:

• Short Prompt Memory (SPM): Maintains a First-In-First-Out (FIFO) bank of recent prompts and their corresponding optimal CAC scores. These are fused using CAC scores as weights to generate a dynamic short-term prompt ($T^{short}_i$), facilitating agile adaptation to local distribution shifts.
• Long Prompt Memory (LPM): Maintains a global prompt ($T^{long}_i$) updated via Exponential Moving Average (EMA) of the short-term prompts. This serves as a stable semantic anchor, preventing catastrophic forgetting and ensuring stability over time.

3. Densely Supervised Prompt Update (DSPU)

To optimize the prompt embeddings without ground truth, the authors propose a dense supervision strategy:

• Enhanced Pseudo-Labels: The stable long-term prompt ($T^{long}_i$) is applied to the CAC-selected optimal view ($X^*_i$) to generate a soft pseudo-label ($P'_i$).
• Loss Functions:
  • Dense Supervision Loss ($\mathcal{L}_d$): A soft Dice loss encourages predictions from all augmented views (using the short-term prompt) to align with the enhanced pseudo-label.
  • Contrast Loss ($\mathcal{L}_c$): Maximizes the CAC score to reinforce textual-visual semantic alignment.
  • Entropy Loss ($\mathcal{L}_e$): Minimizes pixel-wise entropy to encourage confident predictions.
• Application: $\mathcal{L}_c$ and $\mathcal{L}_e$ are applied only to the optimal view selected by CAC to avoid reinforcing errors on low-quality predictions. The total loss is $\mathcal{L} = \mathcal{L}_d + \alpha\mathcal{L}_c + \beta\mathcal{L}_e$.

Key Contributions

1. Novel Metric (CAC): A text-visual semantic consistency metric that outperforms traditional image-level uncertainty measures (entropy, variance, KLD) in selecting reliable augmented views for medical segmentation.
2. Stable Adaptation Architecture (LSPM): A mechanism that resolves the plasticity-stability dilemma in continual TTA by combining a dynamic short-term memory for local adaptation with a momentum-updated long-term memory for global stability.
3. Efficient Dense Supervision (DSPU): A strategy that generates high-quality pseudo-labels from a stable long-term prompt to supervise the adaptation of short-term prompts, enabling effective learning without ground truth.
4. Parameter Efficiency: The method updates only prompt embeddings (1.02K parameters), which is orders of magnitude fewer than methods updating normalization layers (TENT) or low-rank adapters (TTL).

Experimental Results

The framework was evaluated on two medical datasets: Promise12 (prostate segmentation) and ISIC2018 (skin lesion segmentation).

• Performance: CM-TTA significantly outperformed existing TTA methods (including TENT, TPT, TTL, and ZERO) and the non-adapted SAM3 baseline.
  • Promise12: Achieved a Dice score of 73.96% (vs. 72.05% for the best baseline, TPT) and reduced Average Symmetric Surface Distance (ASSD) to 10.91 pixels.
  • ISIC2018: Achieved a Dice score of 82.34% (vs. 80.10% for the best baseline, ZERO) and reduced ASSD to 9.39 pixels.
• Efficiency: The method added negligible inference time (0.57s vs. 0.31s for unadapted SAM3) and required minimal trainable parameters.
• Ablation Studies: Results confirmed that each component (CAC, LSPM, DSPU) contributes incrementally to performance. CAC was shown to be superior to alternative view selection metrics, and hyperparameters (memory length $L=16$, contrast weight $\alpha=1.0$) were found to be generalizable across datasets.

Significance and Claims

The paper claims that CM-TTA addresses the critical issues of over-confidence and confirmation bias inherent in conventional uncertainty-based TTA when applied to text-driven segmentation models in medical domains. By leveraging multi-modal semantic consistency (via CAC) and navigating the plasticity-stability trade-off (via LSPM), the framework establishes an efficient, self-correcting adaptation loop. The authors position this work as a significant step toward deploying foundation models in real-world clinical settings where data distribution shifts are common and annotations are unavailable during testing. The authors note that extending this framework to 3D volumetric medical segmentation is a direction for future work.