A Large-Scale Neutral Comparison Study of Survival Models on Low-Dimensional Data

This large-scale neutral benchmark study evaluates 19 survival models across 34 low-dimensional datasets and concludes that, despite the superior performance of some complex machine learning methods in specific metrics, the standard Cox Proportional Hazards model remains a robust and sufficient choice for most practitioners in this setting.

The Gist

Imagine you are a doctor trying to predict how long a patient might live after a diagnosis. You have a list of patients, some of whom have passed away (the "event"), and some who are still alive but you've lost track of them or the study ended (this is called "censoring"). Your goal is to build a crystal ball that predicts survival time based on their medical features.

For decades, doctors have used a classic, reliable tool for this: the Cox Proportional Hazards model. It's like a trusted, old-fashioned Swiss Army knife—simple, sturdy, and gets the job done.

In recent years, however, a new generation of "super-tools" has arrived: Machine Learning (ML) algorithms. These are like high-tech, laser-guided drones. They are complex, can handle massive amounts of data, and promise to see patterns the old tools miss.

The Big Question: Do these fancy, complex drones actually work better than the trusty Swiss Army knife for the average doctor dealing with standard medical data?

This paper is the ultimate "neutral taste test" to find out.

The "Taste Test" Setup

The authors, a team of statisticians and data scientists, didn't just pick one dataset or one model. They set up a massive, fair competition:

1. The Contestants: They gathered 19 different models.
   • The Classics: The old-school statistical methods (like the Cox model).
   • The Moderns: The fancy machine learning methods (like Random Forests, Boosting, and Neural Networks).
2. The Arena: They tested these models on 34 different real-world datasets (like patient records from hospitals).
3. The Rules:
   • No Cheating: They treated every model exactly the same. They didn't give the "fancy" models a head start or the "old" models a handicap.
   • Tuning: They spent time adjusting the settings (hyperparameters) for every single model to make sure each one was performing at its absolute best.
   • The Scorecard: They measured success in two ways:
     • Ranking: Can the model tell who will die sooner and who will live longer? (Discrimination).
     • Accuracy: Is the predicted probability of survival actually correct? (Calibration/Scoring).

The Results: The Underdog Wins (Again)

After running thousands of simulations and crunching the numbers, the results were surprising to many in the tech world but comforting to many in the medical world:

The "Fancy" Drones didn't beat the "Old" Knife.

• The Verdict: While some of the complex machine learning models did slightly better in specific cases, none of them significantly outperformed the classic Cox Proportional Hazards model when averaged across all the datasets.
• The "Best" of the Rest: A few complex models (like Oblique Random Survival Forests) came close, but they didn't win the race.
• The Takeaway: For standard, low-dimensional data (where you have more patients than variables, which is common in medicine), the simple, old-school Cox model is still the champion. It is robust, easy to interpret, and just as accurate as the complex alternatives.

Why This Matters: The "Over-Engineering" Trap

Think of it like this: If you need to drive to the grocery store, you don't need a Formula 1 race car. A reliable sedan (the Cox model) gets you there just as fast, costs less to maintain, and is easier to drive.

The paper warns practitioners against "over-engineering." Just because you can use a complex AI model doesn't mean you should.

• Complexity Cost: The fancy models are harder to tune, take longer to run, and are often "black boxes" (you can't easily explain why they made a prediction).
• The Recommendation: Start with the simple Cox model. Only switch to the complex machine learning tools if you have a very specific, difficult problem that the simple model can't solve.

In a Nutshell

This study is a massive, fair comparison that says: "Don't throw away your old tools just because new ones look shinier."

For most survival analysis problems in medicine and business, the classic, simple statistical methods remain the gold standard. The complex machine learning models are powerful, but for this specific job, they aren't necessarily better. They are the Ferrari in a traffic jam; the old sedan is still the most efficient way to get to work.

Technical Summary

1. Problem Statement

Survival analysis is critical for predicting time-to-event outcomes in fields like medicine and finance, particularly when data is right-censored (the event has not occurred for some subjects by the end of the study). While numerous machine learning (ML) methods (e.g., Random Survival Forests, Gradient Boosting, Deep Learning) have been proposed to outperform classical statistical models, existing benchmark studies suffer from significant limitations:

• Small Scale: Previous studies often use very few datasets.
• Lack of Neutrality: Many studies focus on validating a specific new model rather than comparing a broad range of methods fairly.
• Insufficient Tuning: Comparisons often rely on default hyperparameters rather than rigorous optimization.
• Narrow Evaluation: Many studies focus solely on discrimination (ranking ability) while ignoring calibration (probability accuracy) or overall predictive performance.

This paper addresses the gap by conducting the first large-scale, neutral benchmark specifically for low-dimensional (features $p <$ observations $n$), single-event, right-censored survival data.

2. Methodology

Study Design & Neutrality

The study adheres to the "neutral comparison study" guidelines by Boulesteix et al. (2013):

• Fair Setup: The authors contacted maintainers of all 19 evaluated models to discuss hyperparameter configurations, ensuring no bias in model setup.
• Scope: The study focuses on the status quo of existing methods rather than proposing a new algorithm.
• Data: 34 publicly available real-world datasets were selected. Inclusion criteria required:
  • Right-censored, single-event data.
  • Low-dimensional ($p < n$).
  • At least 100 observed events.
  • No competing risks or recurrent events.

Models Evaluated (19 Learners)

The study compares a diverse range of algorithms, categorized as:

• Baselines: Kaplan-Meier (KM), Nelson-Aalen (NEL), Akritas Estimator (AK).
• Classical/Parametric: Cox Proportional Hazards (CPH), Regularized CPH (GLMN), Penalized CPH (Pen), Parametric AFT, Flexible Splines.
• Tree-Based: Random Survival Forests (RFSRC, RAN), Conditional Inference Forest (CIF), Oblique Random Survival Forest (ORSF), Relative Risk Tree (RRT).
• Boosting: Model-Based Boosting (Cox/AFT objectives), CoxBoost, XGBoost (Cox/AFT objectives).
• Other: Survival SVM (SSVM).

Note: Deep Learning methods (e.g., DeepSurv, DeepHit) were excluded due to unstable implementations and high computational complexity within the benchmark framework.

Experimental Protocol

• Resampling: Nested repeated cross-validation (3 outer folds, 3 inner folds). Outer folds repeated 5–10 times depending on event count to ensure stability.
• Tuning: Bayesian optimization was used for hyperparameter tuning.
  • Tuning Measures: Models were tuned separately for Harrell's C-index (discrimination) and Integrated Survival Brier Score (ISBS) (overall predictive performance/calibration).
  • Effort: Each tunable parameter received 50 optimization iterations (or exhaustive search for small spaces).
• Pre-processing: Standardization and dummy encoding were applied only where technically required or standard for the specific algorithm.
• Evaluation Metrics:
  • Primary: Harrell's C (discrimination) and ISBS (scoring rule/calibration).
  • Secondary: Uno's C, Integrated Survival Log-Likelihood (ISLL), D-Calibration, and van Houwelingen's $\alpha$.
• Statistical Analysis: Global Friedman rank sum tests followed by post-hoc Bonferroni-Dunn tests to determine significant differences between models, using CPH as the reference.

3. Key Contributions

1. Scale and Scope: The largest neutral benchmark to date for low-dimensional survival data, covering 34 datasets and 19 models.
2. Rigorous Tuning: Unlike previous studies, this benchmark optimizes models for both discrimination and overall predictive ability, ensuring a fair comparison of potential.
3. Open Science: All code, data, hyperparameter search spaces, and results are publicly available via GitHub and OpenML, setting a new standard for reproducibility in survival analysis.
4. Comprehensive Evaluation: The study evaluates not just ranking (C-index) but also calibration and overall predictive accuracy (ISBS), providing a holistic view of model performance.

4. Results

Discrimination (Harrell's C)

• Top Performers: Oblique Random Survival Forests (ORSF), Likelihood-based Boosting (CoxBoost), and Parametric AFT models achieved the highest average ranks.
• Comparison to CPH: While ML methods (trees, boosting) and parametric models showed superior average ranks, no method significantly outperformed the standard Cox Proportional Hazards (CPH) model after statistical correction.
• Baselines: All models significantly outperformed non-parametric baselines (KM, NEL, AK).

Overall Predictive Performance (ISBS)

• CPH Dominance: When evaluated on ISBS (which penalizes poor calibration), the CPH model remained robust.
• Outliers: ORSF and CoxBoost performed slightly better than CPH but were not statistically significant in all contexts.
• Underperformers: XGBoost (Cox objective) and Regularized CPH (GLMN) were significantly outperformed by CPH in terms of overall performance.
• Calibration: Many ML methods that performed well on discrimination (ranking) showed poorer calibration (probability estimation) compared to CPH, leading to lower ISBS scores.

Calibration

• D-Calibration & van Houwelingen's $\alpha$: Results were mixed and dataset-dependent.
• Observations: KM, NEL, and Penalized CPH showed good calibration. Conversely, XGBoost (Cox) and the Akritas estimator showed poor calibration.
• Implication: High discrimination does not guarantee well-calibrated probability estimates.

5. Significance and Conclusions

Main Conclusion:
For low-dimensional, right-censored survival data, the Cox Proportional Hazards (CPH) model remains the most robust and sufficient choice for practitioners. Despite the theoretical advantages of complex ML methods, they do not provide a statistically significant improvement in predictive performance when properly tuned.

Practical Recommendations:

• Practitioners should start with the CPH model due to its interpretability, computational efficiency, and robustness.
• The additional computational cost and loss of interpretability associated with complex ML methods (like Deep Learning or complex ensembles) are generally not justified for standard low-dimensional survival tasks unless specific domain knowledge suggests non-linear interactions that CPH cannot capture.
• If ML methods are used, CoxBoost and Oblique Random Survival Forests are the most promising alternatives, though they require careful tuning.

Future Work:
The authors suggest extending this benchmark to more complex settings (competing risks, left-censoring, high-dimensional data) as software support and evaluation measures for these scenarios mature.

This study fundamentally challenges the assumption that "more complex is better" in survival analysis, reinforcing the enduring value of classical statistical methods in standard settings.