Patient-level coronary artery disease classification from cardiac MRI using out-of-fold heart segmentation and same-slice region controls

Technical Summary: Patient-level Coronary Artery Disease Classification from Cardiac MRI

Problem Statement
The study addresses critical methodological flaws in machine learning applications for medical imaging, specifically regarding data leakage and "shortcut learning." Using the publicly released CAD Cardiac MRI Dataset (Kaggle), the author identified a structural discrepancy: the dataset contained 63,425 JPEG images but only 30 independent patient folders. Previous analyses often treated individual slices as independent samples, leading to severe data leakage where slices from the same patient appeared in both training and testing sets, artificially inflating performance metrics. Furthermore, classifiers may learn non-clinical shortcuts (e.g., scanner style, JPEG export geometry, padding, or extra-cardiac anatomy) rather than disease-specific cardiac features. The study aims to determine if automatically segmented cardiac regions of interest (ROIs) provide superior patient-level discrimination compared to full images or extra-cardiac regions when rigorous patient-level validation and out-of-fold segmentation are enforced.

Methodology
The research employed a retrospective secondary analysis with a strict patient-level cross-validation framework:

• Data Structure & Audit: The 63,425 images were reorganized into 30 patient units (16 Normal, 14 Sick). A comprehensive quality audit and manual target review were conducted, expanding the manual annotation set to 5,097 targets (3,949 heart-present, 1,148 no-heart).
• Out-of-Fold Segmentation: A patient-level cross-fitted 2.5D Attention U-Net was trained on five folds. Crucially, for every patient, the segmentation model was trained on all other patients, ensuring that the resulting cardiac masks for any given patient were truly "out-of-fold" (OOF). This prevented the classifier from seeing the patient's own segmentation during training.
• Representation Construction: Three primary representations were generated from the exact same 50,735 attention-eligible slices:
  1. Full Image (B2): The entire image.
  2. Automatic Cardiac ROI (AU1): The region defined by the OOF segmentation mask.
  3. Complement (C1): The exact pixels outside the ROI.
  • A frozen ImageNet-pretrained EfficientNet-B0 encoded these representations into embeddings.
• Patient Aggregation & Classification: Slice embeddings were averaged within series and then across series to create a single patient vector. A class-balanced logistic regression classifier (with internal cross-validation for hyperparameter tuning) was applied to these patient-level vectors.
• Statistical Rigor: Performance was evaluated using patient-level bootstrap intervals (2,000 resamples) on the 30 patients. Secondary analyses included a cross-class matched cohort (pairing Sick/Normal images based on acquisition descriptors) and a manual-positive subset comparing automatic vs. researcher-created masks.

Key Results

• Segmentation Quality: The OOF segmentation achieved a patient-balanced Dice score of 0.809 on heart-present targets. Notably, 73.4% of "no-heart" targets received empty masks, though a failure mode persisted where ~26% of non-heart frames still generated false-positive masks.
• Classification Performance (Primary Cohort):
  • Automatic ROI (AU1): Achieved an AUC of 0.938 (95% CI: 0.847–1.000).
  • Full Image (B2): Achieved an AUC of 0.830.
  • Complement (C1): Achieved an AUC of 0.772.
  • The paired difference between AU1 and B2 was 0.107 (95% CI: 0.009–0.235), and between AU1 and C1 was 0.165 (95% CI: 0.029–0.344), both excluding zero.
• Matched Cohort: In the secondary matched analysis, the automatic ROI (AU2) maintained an advantage (AUC 0.942) over the full image (0.821) and complement (0.741).
• Automatic vs. Manual: On the 3,949 manually annotated images, the automatic ROI (AU3) achieved an AUC of 0.973, compared to 0.951 for the manual ROI (M1). The paired difference (-0.022) included zero, indicating no statistically significant difference in this sample.
• Residual Shortcut Signal: Despite the ROI advantage, the complement regions (pixels outside the heart) retained substantial predictive power (AUCs ranging from 0.741 to 0.835), suggesting that dataset-specific artifacts (e.g., acquisition style, padding, or extra-cardiac anatomy) still correlate with the disease labels.

Significance and Claims
The paper positions itself as a methodological proof of concept rather than a clinically ready solution. Its primary contributions are:

1. Leakage Control: It demonstrates the necessity of treating the patient, not the slice, as the independent unit of analysis. By enforcing patient-level folds and OOF segmentation, the study provides a more realistic estimate of model performance, avoiding the over-optimism common in image-level evaluations.
2. ROI Utility: It provides evidence that, within this specific dataset, localizing the cardiac region improves patient-level discrimination compared to using the full image or extra-cardiac regions.
3. Shortcut Detection: The study highlights that even with ROI restriction, significant predictive information remains in the "complement" (non-cardiac) regions. This argues against clinical overinterpretation and suggests that models may still be exploiting dataset-specific shortcuts (e.g., JPEG export patterns or scanner metadata) rather than purely learning cardiac pathology.
4. Reproducibility Framework: The author implemented a robust pipeline with semantic fingerprints and atomic writes, ensuring that changes in manual annotations triggered a complete recomputation of all dependent stages (segmentation, feature extraction, and classification), thereby ensuring the results reflect the updated data state.

Limitations and Scope
The author explicitly states that the findings are limited by the small sample size (30 patients), the lack of external validation, the use of JPEGs instead of DICOMs (missing metadata), and the fact that outcome-informed matching was used in secondary analyses. Consequently, the study concludes that while cardiac localization is promising, the presence of residual shortcut signals and the lack of independent validation mean the model is not ready for clinical use. The work serves primarily to validate a rigorous, leakage-controlled methodology for re-evaluating public medical imaging datasets.