A reproducible and extensible framework for benchmarking competing risks survival models

Technical Summary: A Reproducible and Extensible Framework for Benchmarking Competing Risks Survival Models

Problem Statement

Survival analysis in healthcare often involves competing risks (CR), where the occurrence of one event (e.g., cancer death) precludes the occurrence of others (e.g., cardiovascular death). While numerous statistical and machine learning methods have been proposed to handle CR, their systematic evaluation and clinical adoption remain limited. Current literature suffers from three primary gaps:

1. Lack of Reproducible Benchmarks: Studies often use disparate datasets, unreported preprocessing steps, or inconsistent evaluation metrics (frequently limited to discrimination), leading to over-optimism and difficulty in comparing new methods.
2. Limited Software Integration: Novel methods are often implemented as ad-hoc code without integration into broader machine learning frameworks, hindering accessibility.
3. Interpretability Challenges: Unlike regression models that provide direct coefficient estimates, complex machine learning models (e.g., deep learning) require post-hoc interpretability steps, which are often not standardized for CR settings.

Furthermore, treating CR events as censored violates the non-informative censoring assumption, leading to overestimated risks and potential algorithmic bias, particularly in ageing populations with multimorbidity.

Methodology

The authors developed an open-source, reproducible benchmarking pipeline designed to evaluate CR models across multiple datasets under a unified experimental setting. The framework is implemented in both R and Python, ensuring interoperability.

1. Data and Models:

• Datasets: The study utilized five datasets with varying sample sizes and event rates (PBC, HD, METABRIC, Framingham, and a subset of SEER Breast Cancer), ensuring a minimum competing risk event rate of ~10%.
• Models: A diverse set of six methods was benchmarked, ranging from classical regression to deep learning:
  • Regression-based: Cause-specific Cox PH (csCPH), Fine-Gray regression (FGR), Penalized Proportional Subdistribution Hazard (FGRP).
  • Tree-based: Random Survival Forests (RSF).
  • Deep Learning: DeSurv (continuous time) and DeepHit (discrete time).
• Validation: A nested cross-validation approach was employed to prevent data leakage. The outer loop handled model evaluation, while the inner loop managed hyper-parameter tuning.
• Missing Data: Strategies included complete case analysis, univariate imputation (median/mode), and Multiple Imputation by Chained Equations (MICE), all applied within the CV folds to avoid leakage.

2. Performance Metrics:
The framework evaluates four complementary aspects of predictive performance:

• Calibration: Visual plots, Integrated Calibration Index (ICI), and Observed-to-Expected (O/E) ratios, using pseudo-observations to handle censoring.
• Discrimination: Time-dependent AUC (tdAUC) for fixed horizons and C-index variants ($C_{td}$ and $C_{\tau}$) for global assessment. The latter utilizes Restricted Mean Time Lost (RMTL) to summarize CIFs.
• Overall Prediction Error: Weighted Brier Score (BS) and Integrated Brier Score (IBS).
• Clinical Utility: Decision Curve Analysis (DCA) to assess net benefit against "treat-all" and "treat-none" strategies.

3. Interpretability:

• Model-specific: Regression coefficients for parametric models.
• Model-agnostic:
  • Permutation Importance: Measured via IBS degradation.
  • CifSHAP(t): A novel extension of SurvSHAP(t) to CR settings. This method decomposes the cause-specific Cumulative Incidence Function (CIF) over time to provide time-dependent, model-agnostic feature contributions.

4. Experimental Design:
Hyper-parameter tuning was standardized across models using a grid search optimized for IBS (minimizing overall prediction error) to ensure fair comparisons, rather than model-specific loss functions. Deep learning models were trained with a fixed epoch budget (300) to avoid biasing tuning toward specific loss functions via early stopping.

Key Results

• Prediction Consistency: Classical regression models (csCPH, FGR, FGRP) showed near-perfect agreement in individual-level predictions. Machine learning models (RSF, DeSurv, DeepHit) produced more distinct risk assignments, with DeepHit showing the lowest correlation to regression-based models.
• Performance Variability: No single model dominated across all metrics and datasets.
  • DeSurv often outperformed others in global measures (IBS, $C_{\tau}$) on larger datasets like METABRIC.
  • FGRP (Penalized Fine-Gray) remained highly competitive, particularly on larger datasets (Framingham, SEER), often leading in discrimination ($C_{td}$).
  • DeepHit frequently ranked as the worst performer in this benchmark, particularly regarding calibration at later time points.
• Metric Sensitivity: Model rankings were highly sensitive to the chosen metric. For instance, a model with the lowest IBS did not necessarily have the highest C-index. This underscores the risk of "metric-hacking" if evaluation criteria are not pre-specified based on clinical goals.
• Interpretability: Both permutation importance and CifSHAP(t) yielded highly consistent covariate rankings (e.g., Chemotherapy, MKI67, PGR were top predictors in METABRIC). CifSHAP(t) further revealed that covariate contributions (e.g., chemotherapy) could vary in sign and magnitude depending on the observed event type (cause 1 vs. cause 2) and time.
• Optimization Impact: Tuning hyper-parameters based on IBS produced more balanced models (better calibration and discrimination) compared to tuning based on model-specific loss functions or $C_{td}$, which often sacrificed calibration for discrimination.
• Runtime: Regression-based methods were computationally efficient. Machine learning methods, particularly RSF and DeSurv, incurred significantly higher computational costs, with RSF showing the greatest burden due to ensemble tree fitting.

Significance and Claims

The paper positions this work not as a definitive ranking of "best" models, but as a reproducible and extensible foundation for the systematic evaluation of competing risks methods.

• Standardization: The framework addresses the lack of unbiased comparison studies by providing a common experimental setup, consistent preprocessing, and a unified set of evaluation metrics.
• Clinical Relevance: By moving beyond discrimination to include calibration, overall error, and clinical utility (DCA), the framework aligns model evaluation with practical clinical decision-making needs.
• Interpretability Innovation: The introduction of CifSHAP(t) extends model interpretability to time-dependent, cause-specific contributions, filling a gap in understanding how covariates drive risk in CR settings.
• Practical Guidance: The authors emphasize that simpler regression-based methods (like FGRP) often achieve competitive performance with lower computational costs, suggesting they should serve as strong baselines. They also highlight that hyper-parameter tuning criteria (e.g., IBS vs. Loss) critically influence model behavior and rankings, urging transparency in reporting tuning strategies.

The authors conclude that while their findings are conditional on the specific datasets and setup used, the framework facilitates the development, validation, and adoption of future CR prediction models by providing a robust, open-source tool for developers and practitioners.