Dual-Adaptive SAM3: Hierarchical Routing over Low-Rank Expert Layers for Parameter-Efficient Medical Image Segmentation

Technical Summary: Dual-Adaptive SAM3 (DA-SAM3)

Problem Statement
The Segment Anything Model with Concepts (SAM3) offers a promising paradigm for open-vocabulary medical image segmentation through natural language interaction. However, adapting such large-scale vision-language foundation models to the specialized, heterogeneous domain of medical imaging presents a critical efficiency-specialization trade-off. Naive full fine-tuning is parameter-inefficient and risks catastrophic forgetting of pre-trained knowledge. Conversely, standard Mixture-of-Experts (MoE) approaches, while capable of specialization, introduce prohibitive computational and memory overhead, limiting their scalability in resource-constrained clinical settings. Existing Parameter-Efficient Fine-Tuning (PEFT) methods often apply static, uniform adaptations that fail to address the dynamic and hierarchical nature of medical concept understanding.

Methodology
The authors propose Dual-Adaptive SAM3 (DA-SAM3), a framework designed to achieve high segmentation accuracy with extreme parameter efficiency. The approach integrates a hierarchical MoE adaptation strategy into the SAM3 architecture, specifically replacing Feed-Forward Network (FFN) blocks within the model's fusion module at critical depths ($L/6, L/4, L/2$). The framework consists of three primary components:

1. Frozen Encoders: The image encoder ($f_v$) and text encoder ($f_t$) remain entirely frozen to preserve pre-trained general knowledge. Only the newly inserted Dual-Adaptive MoE layers and Layer Normalization (LN) parameters within the fusion module are trainable.
2. Dynamic Expert Router (DER): This task-aware mechanism performs sparse, token-wise expert selection. It jointly reasons about visual content and textual concept prompts to generate gating weights. The router utilizes a global domain-context vector ($h_{ctx}$), derived from the interaction between concept tokens and pooled visual features, alongside local token features and the original concept semantics. This mimics a context-sensitive clinical consultation process, selecting the top-$k$ experts (where $k=2$) for each token.
3. Decomposed Parameterized Experts (DPE): To address parameter efficiency, each expert is not a full dense network but is decomposed into a shared, frozen base weight ($W_0$ inherited from SAM3) and a lightweight, trainable low-rank delta ($\Delta W_i = A_i B_i^\top$). This design reduces MoE parameter overhead by over 80% compared to standard MoE implementations.

Hierarchical Specialization Strategy
The framework employs a hierarchical placement of MoE layers to emulate a coarse-to-fine clinical reasoning process:

• Early Stage ($L/6$): Focuses on coarse global alignment, aligning low-level visual primitives with input concepts.
• Mid Stage ($L/4$): Dedicated to semantic structure identification, integrating multi-modal features to resolve morphological variations and distinguish adjacent structures.
• Late Stage ($L/2$): Focused on precision boundary refinement, operating on high-resolution feature maps to sharpen boundaries and resolve regional ambiguities.

The model is trained using a two-stage strategy: a warm-up stage for expert specialization with load balancing, followed by a fine-tuning stage for routing calibration on broader datasets to enhance open-vocabulary generalization.

Key Contributions

1. Novel Framework: DA-SAM3 is the first framework to integrate dynamic multimodal routing with parameter-efficient expert decomposition for adapting vision-language foundation models to medical imaging.
2. Dual-Adaptive Mechanisms: The introduction of the Dynamic Expert Router (DER) for context-aware sparse selection and Decomposed Parameterized Experts (DPE) for drastic parameter reduction.
3. Hierarchical Design: A specialized strategy where experts at different network depths learn to perform distinct tasks (alignment, identification, refinement), mirroring clinical reasoning.

Experimental Results
Extensive experiments were conducted on four public medical segmentation benchmarks: Synapse (abdominal CT), MMWHS (cardiac CT), BTCV (abdominal CT), and ACDC (cardiac MRI).

• Performance: DA-SAM3 achieves state-of-the-art results, outperforming current methods by approximately 5% in key metrics. On the Synapse dataset, it improved the Dice Similarity Coefficient (DSC) to 85.12% (a 4.37% gain over the SAM3 baseline) and reduced the Hausdorff Distance (HD) from 15.120mm to 8.425mm.
• Comparisons: The model consistently outperforms fully fine-tuned SAM3, standard MoE baselines, and other task-specific architectures (e.g., nnU-Net, Swin-UNETR) across all datasets.
• Efficiency: The DPE design reduces MoE parameter overhead by over 80% while maintaining or exceeding the accuracy of standard fine-tuning.
• Ablation Studies: Removing the DER (replacing with random routing) or the DPE (using full experts) resulted in significant performance drops, validating the necessity of both the dynamic routing and the parameter-efficient decomposition.

Significance
The paper claims that DA-SAM3 provides a practical pathway for deploying versatile, generalist AI models in clinical practice. By successfully balancing extreme parameter efficiency with high segmentation accuracy, the framework addresses the critical need for adaptable, resource-efficient medical AI. The results suggest that mimicking clinical decision-making processes through hierarchical, dual-adaptive specialization allows large foundation models to be effectively specialized for diverse medical concepts without the computational burden of full fine-tuning or standard MoE.