CTSCAN: Evaluation Leakage in Chest CT Segmentation and a Reproducible Patient-Disjoint Benchmark

1. Problem Statement

The paper addresses a critical flaw in the evaluation of chest CT segmentation models: evaluation leakage caused by improper data partitioning.

• The Issue: Many existing chest CT segmentation pipelines export volumetric studies into per-slice PNG images. When researchers partition data for training, validation, and testing, they often split these individual slices randomly rather than grouping them by patient case (study ID).
• The Consequence: This "slice-mixed" approach allows slices from the same patient to appear in both the training and test sets. Consequently, models appear to generalize well because they are effectively memorizing patient-specific anatomical features rather than learning to segment pathology across different patients.
• The Impact: Reported performance metrics (Dice Score, IoU) are artificially inflated, leading to misleading scientific conclusions about model efficacy and generalizability.

2. Methodology

The authors introduce CTSCAN, a reproducible benchmark and research stack designed to enforce patient-disjoint evaluation (where no patient appears in more than one split).

• Dataset Composition:
  • Aggregates 89 cases from three public sources: PleThora (78 cases), MedSeg SIRM (10 cases), and LongCIU (1 case).
  • Total of 10,423 slices.
  • Classes: Four classes (Background, Ground Glass, Consolidation, Pleural Effusion). The dataset is highly imbalanced, with background pixels dominating (99.6%) and pleural effusion being the most frequent foreground class.
• Experimental Protocols:
  The study compares two distinct partitioning protocols using the same underlying data and model architecture:
  1. Slice-Mixed Protocol: Slices are pooled into a single list and split randomly (the common, flawed approach).
  2. Patient-Disjoint Protocol: Slices are grouped by Case ID before splitting, ensuring zero overlap of patients between train, validation, and test sets.
• Model Configuration:
  • Architecture: FPN (Feature Pyramid Network) with an EfficientNet-B0 encoder.
  • Loss: Dice-CE (Dice Loss + Cross-Entropy).
  • Hyperparameters: Fixed across protocols (Image size 256, Batch size 8).
  • Validation: A multi-seed protocol sweep (3 random seeds, 12 epochs each) to ensure statistical robustness.
• Control Variables: The authors also tested a U-Net architecture and ablated loss functions (Lovász-Softmax) and samplers (Rare-FG) to ensure the results were not specific to one model or training tweak.

3. Key Contributions

1. Exposure of Evaluation Leakage: The paper quantifies that the standard slice-mixed protocol induces near-complete case reuse (e.g., 89/89 cases in training also appear in testing), rendering standard metrics unreliable.
2. CTSCAN Benchmark Release: A public, reproducible artifact containing:
   • Deterministic, patient-disjoint split manifests.
   • Explicit overlap auditing scripts.
   • A scripted multi-seed protocol sweep.
   • A public leaderboard that ranks models only under patient-disjoint evaluation.
3. Executable Evidence: Instead of anecdotal claims, the authors provide a full research stack (code, scripts, and presets) that allows the community to reproduce the "leakage vs. generalization" gap immediately.

4. Key Results

The study demonstrates that the choice of evaluation protocol fundamentally alters the scientific conclusion regarding model performance.

• Performance Collapse under Patient-Disjoint Evaluation:
  • Slice-Mixed (Flawed): Foreground Dice = 0.6665 ± 0.0067; Foreground IoU = 0.5031 ± 0.0070.
  • Patient-Disjoint (Correct): Foreground Dice = 0.2066 ± 0.0509; Foreground IoU = 0.1181 ± 0.0313.
  • Impact: Removing patient reuse caused an absolute drop of 0.4599 in Dice (a 69.00% relative reduction) and a 76.52% relative reduction in IoU.
• Architecture Agnosticism: The leakage effect is not specific to FPN. A shared U-Net control showed a similar collapse (Dice dropped from 0.7098 to 0.2472).
• Training Tweaks Cannot Compensate: Ablation studies (Rare-FG sampling, Lovász-CE loss) improved scores slightly within each protocol but failed to close the massive gap between the slice-mixed and patient-disjoint regimes.
• Per-Class/Per-Source Analysis: The inflation was consistent across all classes (Ground Glass, Consolidation, Pleural Effusion) and data sources (PleThora, MedSeg SIRM, LongCIU).

5. Significance and Implications

• Re-evaluation of State-of-the-Art: The paper argues that many previously reported "strong" chest CT segmentation results are likely artifacts of data leakage. The true generalization capability of current models is significantly lower than reported.
• Standardization of Benchmarks: CTSCAN establishes a new standard for medical imaging benchmarks: patient-disjoint splitting must be the default, not an optional step.
• Scientific Rigor: It highlights that in medical imaging, where data is often limited and patient-specific features are strong, failing to isolate patients in the test set leads to memorization rather than learning.
• Actionable Tooling: By providing the code and manifests, the authors enable the community to audit existing models and build future research on a foundation of honest, reproducible evaluation.

In conclusion, CTSCAN demonstrates that protocol choice changes the absolute scientific conclusion. The paper serves as a corrective measure, shifting the focus from "tuning models to high numbers on flawed splits" to "measuring what signal survives under strict patient-disjoint evaluation."