RobustMedSAM: Degradation-Resilient Medical Image Segmentation via Robust Foundation Model Adaptation

1. Problem Statement

Medical image segmentation is vital for clinical workflows but suffers from reliability issues when images are degraded by real-world artifacts such as noise, blur, motion, and modality-specific distortions (e.g., Rician noise in MRI, speckle in ultrasound).

• The Gap: Existing solutions typically address either medical domain adaptation (e.g., MedSAM, which performs well on clean medical data) or corruption robustness (e.g., RobustSAM, which handles natural image corruptions but lacks medical priors).
• The Challenge: No unified framework currently exists that jointly optimizes for medical-domain specificity and resilience to image degradation. Furthermore, standard models like SAM and MedSAM often fail significantly when presented with corrupted inputs or when using point prompts instead of bounding boxes.

2. Methodology: RobustMedSAM

The authors propose RobustMedSAM, a framework based on the hypothesis that medical priors and degradation robustness are concentrated in complementary modules of the Segment Anything Model (SAM) architecture. Instead of training a new model from scratch, they employ a module-wise checkpoint fusion strategy.

A. Core Architecture & Initialization

The model utilizes a shared ViT-B backbone but initializes specific components from different pre-trained models:

• Image Encoder ($\mathcal{E}$) & Prompt Encoder ($\mathcal{P}$): Initialized from MedSAM. This preserves the anatomical representations and modality-specific feature statistics learned from large-scale medical supervision.
• Mask Decoder ($\mathcal{D}$): Initialized from RobustSAM. This module is pre-trained to handle diverse corruptions and includes anti-degradation mechanisms.

B. Training Strategy

The framework employs a clean-degraded pair training approach with a specific focus on parameter efficiency:

1. Data Augmentation: For every clean medical image ($x_c$), a degraded counterpart ($x_d$) is generated using a predefined distribution of 12 corruption types (e.g., noise, blur, motion artifacts) across 6 imaging modalities.
2. Freezing Strategy: The MedSAM image and prompt encoders are frozen to preserve medical priors. Only the RobustSAM-derived mask decoder is fine-tuned.
3. Loss Functions:
   • Segmentation Loss: Supervised only on the degraded branch ($\hat{m}_d$) using Dice and Focal losses to directly optimize robustness.
   • Consistency Regularization: Enforces consistency between the clean and degraded branches via Mean Squared Error (MSE) on intermediate mask features ($\mathcal{L}_{mfc}$) and refined output tokens ($\mathcal{L}_{tc}$). This ensures the decoder learns degradation-invariant representations.
4. SVD-Based Variant: An optional, parameter-efficient adaptation is explored where Singular Value Decomposition (SVD) is applied to the encoder's convolutional weights. Only the singular values are updated while the orthogonal bases remain frozen.

3. Key Contributions

1. Module-Wise Fusion: Demonstrates that initializing the encoder from a medically adapted model and the decoder from a robustness-focused model effectively composes both capabilities, outperforming parent models on average.
2. Decoder-Only Fine-Tuning: Shows that fine-tuning only the decoder with clean-degraded consistency training is sufficient to achieve robust adaptation, preserving clean-image accuracy while significantly improving performance on degraded images.
3. Comprehensive Evaluation: Provides a rigorous evaluation across 35 medical datasets, 6 imaging modalities (Ultrasound, MRI, CT, X-Ray, Endoscopy, Fundus, etc.), and 12 corruption types, including both in-distribution and out-of-distribution (OOD) benchmarks.
4. Analysis of Trade-offs: Characterizes the limits of the approach, noting that while it excels with point prompts and specific corruptions, it faces challenges with topology-sensitive structures (thin vessels) and box-prompt settings where MedSAM's original training bias remains dominant.

4. Experimental Results

Experiments were conducted on the MedSegBench benchmark.

• Performance on Degraded Images:
  • RobustMedSAM achieved a mean Dice score of 0.719 on degraded images.
  • This represents a +0.106 improvement over the baseline SAM (0.613) and a +0.087 improvement over RobustSAM (0.632).
  • In contrast, MedSAM collapsed under point prompting with degraded inputs (Dice $\approx$ 0.201).
• Clean Image Performance:
  • RobustMedSAM maintained performance on clean images comparable to SAM (0.620 vs. 0.621), proving that robustness training did not sacrifice accuracy on high-quality data.
• Prompt Sensitivity:
  • The model showed significant gains with point prompts, where MedSAM typically fails. It saturated in performance with just $K=3$ point prompts.
  • Under box prompts, it remained competitive with MedSAM, though MedSAM retained a slight edge in IoU due to its box-centric training.
• Out-of-Distribution (OOD):
  • The model improved average OOD Dice from 0.455 (SAM) to 0.493, with notable gains on datasets like ISIC 2018 and PanDental.
• Ablation on SVD:
  • The SVD-based encoder adaptation variant showed a negative trade-off: it slightly improved performance on specific encoder-sensitive corruptions (e.g., Poisson noise) but significantly degraded clean-image accuracy (Dice dropped to 0.468), leading the authors to recommend freezing the encoder by default.

5. Significance and Conclusion

RobustMedSAM establishes a practical and effective paradigm for robust medical image segmentation. By leveraging the modularity of foundation models, it successfully bridges the gap between domain adaptation and corruption robustness without the computational cost of full retraining.

• Clinical Relevance: The method ensures reliability in real-world clinical settings where image quality varies due to acquisition noise or patient motion, particularly when using interactive point-based prompting which is common in clinical workflows.
• Methodological Insight: The study confirms that "what to represent" (anatomy) and "how to decode" (robustness) can be decoupled and optimized separately, offering a blueprint for adapting other foundation models to specialized, noisy domains.
• Limitations: The approach is less effective for topology-sensitive structures (e.g., thin vessels) where fine-grained encoder features are critical, suggesting that future work may need to address encoder-level adaptation for such specific cases.