Real-World Data for Predicting Rapid Relapse Triple Negative Cancer: A Study Using NCDB and EHR Data

Technical Summary: Real-World Data for Predicting Rapid Relapse Triple Negative Cancer

Problem Statement
Triple-negative breast cancer (TNBC) is an aggressive subtype accounting for 10–15% of breast cancers but approximately 40% of breast cancer-related deaths. A specific subset of patients, termed rapid relapse TNBC (rrTNBC), experiences recurrence or cancer-related death within 24 months of diagnosis. These patients are disproportionately older, Black, or insured by public programs (e.g., Medicaid), and they frequently fail to receive guideline-concordant treatment (surgery and chemotherapy), which is known to reduce rapid relapse risk by 3x to 4x. While machine learning (ML) has been applied to breast cancer mortality and subtype classification, few studies have focused on predicting rapid relapse using real-world data (RWD) that includes socio-demographic and community-level factors. Existing models often rely on complex biomarkers or demonstrate only modest predictive ability (AUC ~0.77). The objective of this study was to develop a Rapid Relapse Predictive Model (rrPM) capable of identifying high-risk patients early to facilitate timely interventions.

Methodology
The study employed a two-stage approach utilizing data from the National Cancer Database (NCDB) and the Ohio State University (OSU) Cancer Registry.

• Data Sources:

  • Training/Validation: 47,048 TNBC patients from the NCDB (2010–2019) with AJCC stage IB–IIIC and at least 24 months of follow-up.
  • External Testing: 281 TNBC patients from the OSU Cancer Registry (2012–2020) with similar inclusion criteria.
  • Outcome Definition: Rapid relapse (rrTNBC) was defined as all-cause mortality within 24 months of diagnosis.
  • Features: The model utilized a parsimonious set of structured data available at diagnosis, including sociodemographics (age, race, insurance, marital status), community needs (census tract-level education and income), and clinical features (AJCC stage, tumor grade, Charlson Comorbidity Index).

• Model Development:

  • Algorithms: Six ML models were trained: Logistic Regression, Decision Tree, Random Forest, Naïve Bayes, Support Vector Machine (SVM), and XGBoost.
  • Handling Imbalance: Given the minority class prevalence (~14.5% in NCDB), the study employed Synthetic Minority Oversampling Technique (SMOTE) combined with Tomek Links (SMOTE-Tomek) and algorithmic class weighting.
  • Ensemble Strategy: A stacked ensemble was constructed using Logistic Regression, Decision Tree, and Random Forest as base learners, with Logistic Regression serving as the meta-learner.
  • Transfer Learning & Optimization: The ensemble model trained on NCDB data was adapted to the OSU dataset. This involved applying the pre-trained model to local data, followed by transfer learning (fine-tuning) and threshold optimization. The classification threshold was adjusted to maximize the F1 score while maintaining high Positive Predictive Value (PPV) and sensitivity, addressing the specific prevalence and distribution of the local clinical data.

• Evaluation Metrics: Performance was assessed using Sensitivity, Specificity, PPV, Negative Predictive Value (NPV), Accuracy, F1-score, and ROC AUC.

Key Results

• NCDB Baseline Performance: Initial models trained on NCDB data exhibited high NPV (0.86–0.92) but low sensitivity (0.08–0.10) and low PPV (0.22–0.60). While SMOTE and hyperparameter tuning improved sensitivity (up to 0.68 for Naïve Bayes), the models generally struggled to identify true positive cases of rapid relapse, with ROC AUC scores ranging from 0.59 to 0.71. The best-performing meta-learning ensemble on NCDB data achieved a sensitivity of 0.63 and an F1-score of 0.38.
• External Validation (OSU): When the NCDB-trained models were tested directly on the OSU registry without further adaptation, performance remained suboptimal (ROC AUC ~0.71, Sensitivity ~0.58 for the ensemble), confirming a domain shift between national registry data and single-site clinical data.
• Optimized Performance: After applying transfer learning and threshold optimization specifically to the OSU data, the ensemble model's performance improved dramatically. The optimized model achieved:
  • Sensitivity: 0.87
  • Specificity: 0.99
  • PPV: 0.90
  • NPV: 0.98
  • ROC AUC: 0.99
  • Accuracy: 0.98
  • F1-Score: 0.88

Significance and Claims
The paper claims that the primary contribution is the successful development of a high-performing predictive tool for rrTNBC using readily available clinical and socio-demographic data, avoiding the need for complex biomarkers. The study demonstrates that while models trained on large national datasets (NCDB) may not generalize directly to local clinical settings due to distributional shifts, transfer learning combined with threshold optimization can effectively adapt these models to real-world data.

The authors posit that the optimized rrPM, with its high sensitivity and PPV, can serve as a clinical decision support (CDS) tool. By identifying patients at high risk for rapid relapse early, the tool could facilitate targeted patient navigation, ensuring timely access to guideline-concordant treatment and multidisciplinary care, particularly for vulnerable populations. The model is designed to be translatable to Fast Healthcare Interoperability Resources (FHIR) workflows, allowing for potential integration into electronic health records via APIs.

Limitations and Future Directions
The authors explicitly note limitations, including the relatively small sample size of the clinical validation dataset (n=281), which may affect generalizability. They state that the findings require prospective external validation on larger, independent datasets. Furthermore, the paper emphasizes that while the model shows high predictive accuracy, its actual impact on care processes and patient outcomes remains to be determined through implementation studies and prospective trials evaluating its integration into clinical workflows.