RectoMap: a generalizable deep learning tool for rectal cancer and mesorectum MRI segmentation

Technical Summary: RectoMap – A Generalizable Deep Learning Tool for Rectal Cancer and Mesorectum MRI Segmentation

Problem Statement
Accurate segmentation of rectal tumors and the mesorectum on T2-weighted MRI is critical for staging, treatment planning (including radiotherapy and neoadjuvant chemoradiotherapy), and surgical assessment in locally advanced rectal cancer (LARC). However, manual segmentation is time-consuming, operator-dependent, and suffers from inter-observer variability. While deep learning offers automated solutions, existing models often struggle with out-of-distribution (OOD) generalization. Specifically, performance degrades significantly when models trained on data from specific scanner manufacturers are applied to images from unseen vendors due to heterogeneity in acquisition protocols, field strengths (1.5T vs. 3T), and manufacturer-specific biases. Previous studies have shown marked performance drops (e.g., DSC declining from 0.77 to 0.59) when testing on unseen scanners, limiting clinical applicability in real-world, multi-vendor settings.

Methodology
The authors developed RectoMap, a robust, open-source deep learning pipeline designed to address these generalization challenges through a systematic evaluation of architectures, ensembling strategies, and data augmentation.

• Dataset: A retrospective, multi-institutional study involving 226 patients with histologically confirmed LARC. The dataset is highly heterogeneous, comprising scans from 19 distinct MRI scanner models across four vendors (Siemens, Philips, GE, Hitachi) at both 1.5T and 3T.
• Experimental Design: Two evaluation settings were established:
  • In-Distribution (ID): Random split (181 training, 45 testing) where test scanners were present in training.
  • Out-of-Distribution (OOD): Models trained on Siemens, Toshiba, and Philips data (163 subjects) and tested exclusively on unseen GE and Hitachi scanners (63 subjects).
• Architectures: Two complementary 3D models were trained using 5-fold cross-validation:
  • nnUNet: A self-configuring CNN-based framework.
  • U-MambaBot: A hybrid architecture integrating convolutional layers with state-space modeling for long-range context.
• Ensembling Strategies: Four fusion configurations were evaluated to combine predictions from the two architectures and their five folds:
  1. Architecture-wise averaging followed by STAPLE.
  2. Fold-wise best model selection followed by STAPLE.
  3. Two-level STAPLE aggregation: STAPLE applied within each architecture, then the results fused via STAPLE.
  4. Full-model aggregation: All 10 model predictions (5 folds × 2 architectures) fused in a single STAPLE step.
• Data Augmentation: Experiments compared a standard "base" augmentation (default nnUNet) against an extended MRI-specific augmentation scheme. The extended scheme incorporated TorchIO and Global Intensity Non-linear (GIN) transformations to simulate real-world artifacts like bias field inhomogeneity, motion, and intensity non-linearities.
• Metrics: Performance was quantified using the Dice Similarity Coefficient (DSC) and the 95th-percentile Hausdorff Distance (HD95).

Key Results

• In-Distribution (ID) Performance:
  • For rectal cancer, the Two-level STAPLE aggregation (Configuration 3) with extended augmentation achieved the highest mean DSC of 0.767 ± 0.119.
  • For the mesorectum, Full-model aggregation (Configuration 4) yielded the best DSC of 0.794 ± 0.131.
  • MRI-specific augmentation consistently improved mesorectum segmentation.
• Out-of-Distribution (OOD) Performance:
  • Despite the vendor shift, the model demonstrated stable cross-vendor performance.
  • Rectal cancer: Full-model aggregation (without extended augmentation) achieved a DSC of 0.798 ± 0.126.
  • Mesorectum: Full-model aggregation (with extended augmentation) achieved a DSC of 0.776 ± 0.161.
  • Notably, rectal tumor segmentation performance was slightly higher in the OOD setting than ID, attributed to smaller average tumor volumes in the OOD cohort (simpler shapes). Conversely, mesorectum performance was slightly lower in OOD due to larger, more variable volumes in that cohort.
• Failure Analysis: Qualitative review indicated that errors were concentrated in cases with small/annular tumors, collapsed rectal lumens (lack of gel distension), and scanner models underrepresented in training data.

Significance and Contributions
The paper positions RectoMap not merely as a benchmarking exercise but as a ready-to-use, generalizable clinical tool. Its primary contributions include:

1. Robustness to Vendor Shift: By leveraging multi-vendor data and specific augmentation strategies, RectoMap maintains high performance across unseen scanner manufacturers, addressing a critical barrier to clinical deployment.
2. Open-Source Availability: The tool is publicly released (GitLab) with pre-trained weights, allowing direct inference or fine-tuning on local datasets. This provides a reliable baseline even for institutions with limited data.
3. Systematic Validation: The study provides a comprehensive comparison of ensembling strategies and augmentation types, identifying that Configuration 3 (Two-level STAPLE) combined with extended MRI-specific augmentation offers the most robust design for the final tool.
4. Clinical Utility: With an inference time of approximately 82 seconds per volume on an RTX 3090, the pipeline is designed for practical integration into clinical workflows.

The authors conclude that while RectoMap represents a significant step toward robust automated segmentation, limitations remain, including a dataset skewed toward Siemens scanners and the lack of prospective validation regarding its direct impact on treatment planning or patient outcomes. Future work aims to expand the dataset, validate prospectively, and integrate federated learning approaches.