ADEST-U-Net: A Dual-Encoder Architecture for Cross-Organ and Cross-Modality Transfer Learning in Data-Scarce Medical Image Segmentation

Problem Statement

Medical image segmentation, particularly for liver tumor delineation in Computed Tomography (CT), is heavily dependent on large volumes of expert-annotated data, which are often scarce in clinical practice. While transfer learning is a standard strategy to mitigate data scarcity, existing approaches typically focus on same-organ or same-modality adaptation. The paper addresses the specific challenge of cross-organ and cross-modality transfer: adapting representations learned from brain tumor segmentation in Magnetic Resonance Imaging (MRI) to liver tumor segmentation in CT.

The authors note that "naïve" fine-tuning under such severe anatomical (brain vs. liver) and modality (MRI vs. CT) shifts yields limited improvements. A fundamental methodological question arises: under extreme domain mismatch, does single-encoder transfer effectively reuse high-level representations, or does domain-specific interference degrade performance? The study posits that conventional fine-tuning may be insufficient when source and target domains differ simultaneously in anatomy and imaging modality.

Methodology

The study utilizes the 3D-IRCADb-01 dataset, a benchmark for liver and tumor segmentation from contrast-enhanced abdominal CT scans. Due to the sparsity of tumor regions in volumetric scans, the authors decomposed 3D volumes into 2D axial slices, filtering for tumor-containing slices to address class imbalance.

The research is structured into three experimental tracks to evaluate different transfer strategies:

1. Track 1: Baseline Supervised Learning (From Scratch)
   A standard UNet++ architecture was trained from scratch on the liver CT data. This serves as the supervised reference point, as no pretrained model exists for the specific target task.

2. Track 2: Naïve Cross-Organ Transfer
   A UNet++ model pretrained on brain tumor MRI was adapted to the liver CT task. Two strategies were tested:

   • Progressive Fine-tuning: Eleven configurations were evaluated, ranging from freezing only early encoder blocks to freezing the entire network (encoders, bottleneck, and nested decoders).
   • Input Modality Adaptation (IMA): An adapter module was added to the input layer to reduce MRI-CT mismatch before feeding data into the frozen/fine-tuned network.

3. Track 3: Proposed ADEST-U-Net Architecture
   To mitigate the limitations observed in Track 2, the authors propose the Adapted Dual-Encoder Shared-Tail U-Net (ADEST-U-Net).

   • Dual-Encoder Design: The architecture employs two parallel encoders:
     • Encoder A: Initialized with weights from the brain MRI-pretrained UNet++ (with input modality adaptation).
     • Encoder B: A CT-specific encoder trained from scratch.
   • Shared-Tail Refinement: Features from both encoders are concatenated at the bottleneck (resulting in 1024 channels) and passed through a shared refinement module (Conv + BN + ReLU + Dropout) that reduces channels back to 512 before decoding.
   • Design Rationale: This design explicitly separates transferable features (from MRI) from domain-specific features (CT) to prevent negative transfer, while fusing them in a controlled manner before decoding.
   • Training Strategy: A two-phase training process was used: a warm-up phase where Encoder A was frozen, followed by end-to-end fine-tuning.

Loss Function: All models were trained using a composite loss combining Dice Loss and a Shape Prior Loss (derived from the empirical probability of tumor occurrence in the training masks).

Key Results

The models were evaluated on the 3D-IRCADb-01 test set using Dice Similarity Coefficient (DSC), Intersection over Union (IoU), Precision, Recall, and boundary metrics (HD95, ASSD).

• Baseline vs. Naïve Transfer: The baseline UNet++ trained from scratch achieved a test Dice of 0.7020. In contrast, naïve transfer from brain MRI (even with input modality adaptation) performed significantly worse, with the best transfer configuration achieving a Dice of only 0.5383. This confirms that direct single-encoder transfer under extreme domain shifts results in negative transfer.
• ADEST-U-Net Performance: The proposed ADEST-U-Net achieved a test Dice of 0.7189 and an IoU of 0.5611, slightly outperforming the supervised baseline.
• Error Analysis:
  • False Positives: ADEST-U-Net reduced false positives by 67.6% compared to the baseline trained from scratch and by 40.4% compared to frozen-weight transfer.
  • Precision vs. Recall: The baseline model exhibited high recall (0.8712) but low precision (0.5870) due to over-segmentation and extrahepatic false detections. ADEST-U-Net achieved a more balanced trade-off with the highest precision (0.7727) and a recall of 0.6720.
  • Boundary Quality: While the mean Dice scores between the baseline and ADEST-U-Net were not statistically significantly different (p = 0.273), boundary metrics showed marked improvement. The median Hausdorff Distance (HD95) decreased from 220.57 (baseline) to 78.41 (ADEST), indicating smoother, more anatomically coherent predictions.
  • Detection Sensitivity: The baseline detected tumors in 85.96% of slices, whereas ADEST-U-Net detected them in 59.65%. The authors attribute this to ADEST's conservative nature, which suppresses spurious activations at the cost of missing some small lesions.

Significance and Claims

The paper modestly claims that its primary contribution is not achieving state-of-the-art segmentation accuracy (as other architectures on 3D-IRCADb-01 have reported higher tumor Dice scores), but rather providing architectural and empirical insights into cross-organ and cross-modality transfer learning.

The authors conclude that:

1. Direct reuse of pretrained representations in a single-encoder architecture under extreme domain mismatch leads to negative transfer.
2. Architectural mitigation via a dual-encoder design is necessary to separate modality-specific learning from pretrained feature reuse.
3. Modality-aware separation allows for controlled feature interaction, resulting in improved structural plausibility and a significant reduction in false positives, even if it does not immediately maximize the Dice score.
4. The study highlights the limitations of relying solely on overlap-based metrics (like Dice) for small-lesion segmentation, suggesting that boundary regularity and precision are critical indicators of successful cross-domain adaptation in clinical settings.

The work suggests that for cross-organ transfer to be viable, the architecture must explicitly distinguish between transferable and domain-specific information rather than relying on conventional fine-tuning.