📊 statistics

Comparative Review of Modern Competing Risk Methods in High-dimensional Settings

This study provides a comprehensive comparative evaluation of penalized regression, boosting, random forest, and deep learning methods for high-dimensional competing risk analysis, revealing that CoxBoost offers superior false discovery control and stability while highlighting specific strengths and limitations of other approaches across variable selection, accuracy, and calibration.

Original authors: Paul M. Djangang, Summer S. Han, Nilotpal Sanyal

Published 2026-07-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Paul M. Djangang, Summer S. Han, Nilotpal Sanyal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to predict when a suspect will get caught. In the world of medicine and engineering, this "suspect" is a patient or a machine, and the "getting caught" is an event like a disease returning or a part breaking. Usually, detectives just watch for one specific outcome. But in real life, things are messier. A patient might die from a heart attack before their cancer returns, or a machine might rust away before a gear snaps. These are "competing risks": different ways an event can happen that stop each other from occurring. If you ignore the heart attack and only look at the cancer, you might think the cancer is more dangerous than it really is, because you forgot that the heart attack "stole" the patient's time.

To solve this mystery, scientists use special math tools called "survival analysis." But what happens when you have a massive pile of clues—like thousands of genes or sensors—instead of just a few? This is the "high-dimensional" problem. It's like trying to find a single needle in a haystack, except the haystack has millions of needles, and you don't know which ones are real clues and which are just hay. Researchers have built many different "detective kits" (statistical methods) to handle this, but they haven't really tested them all against each other in these messy, crowded situations. This paper steps in to put those kits to the test, running thousands of simulated cases to see which detective is actually the sharpest.

The authors of this study set up a giant digital laboratory to compare four main types of detective kits: Penalized Regression (a method that tries to shrink the list of suspects down to the most likely ones), Boosting (a technique that builds a model step-by-step, learning from its mistakes), Random Forest (a team of decision trees that vote on the answer), and Deep Learning (a complex neural network that tries to mimic the human brain). They created 672 different scenarios, changing the number of patients, the number of clues, how the clues were connected, and how tricky the patterns were. They then watched how well each method could pick out the true clues, guess the right risk, and predict the future without getting confused.

Here is what they found in their simulations. The Boosting method (specifically called CoxBoost) turned out to be the most reliable detective in the crowded rooms. When there were thousands of clues and fewer patients, it was the best at keeping "false alarms" low. It didn't get distracted by the hay; it stuck to the real needles. It also did a great job at ranking patients by risk, telling you who was most likely to have an event first.

On the other hand, the Penalized Regression methods (like LASSO, SCAD, and MCP) were great when the room wasn't too crowded (when there were more patients than clues). They were excellent at giving very accurate probability numbers, telling you exactly how likely an event was to happen. However, when the room got too crowded with clues, they started to get a bit jittery, sometimes picking up too many false leads or becoming unstable.

The Random Forest and Deep Learning methods were the wildcards. They are famous for finding tricky, non-linear patterns—like realizing that a clue only matters if two other clues are present at the same time. While they are powerful, in this specific test, they struggled a bit. Random Forest tended to pick way too many suspects, making it hard to know which ones were actually important. Deep Learning was fast but often gave poor probability estimates, meaning it was bad at saying "there is a 30% chance" versus "there is a 70% chance."

To prove these methods worked in the real world, the team also tested them on a dataset of 214 melanoma (skin cancer) patients with over 47,000 gene clues. The Boosting method picked just one gene, while Random Forest picked over 1,200. Interestingly, both methods found genes that scientists already know are linked to melanoma, but the Boosting method was much more focused, while Random Forest cast a very wide net.

The bottom line of this study is that there is no single "perfect" detective for every crime scene. If you are dealing with a massive amount of data and need to know who is at highest risk without getting false alarms, the Boosting method seems to be the most robust choice. If you have a smaller dataset and need precise probability numbers, the Penalized Regression methods might be better. And while the fancy AI and tree-based methods are great at finding complex patterns, they might need more tuning and bigger datasets to shine as brightly as the more traditional, structured approaches in these high-stakes, high-clue environments. The authors suggest that choosing the right tool depends entirely on the size of your data and what you need to get out of it: a short list of suspects, a precise probability, or a deep understanding of complex interactions.

Technical Summary: Comparative Review of Modern Competing Risk Methods in High-dimensional Settings

Problem Statement
Competing risk analysis addresses scenarios where multiple mutually exclusive events can occur, such that the occurrence of one precludes the others. Traditional survival analysis methods, such as the Kaplan–Meier estimator and the Cox proportional hazards model, often fail in these settings by treating competing events as independent censoring, leading to overestimated survival probabilities. While statistical approaches like Proportional Cause-Specific Hazards (PCSH) and Proportional Subdistribution Hazards (PSDH/Fine–Gray) exist, they are not directly applicable to high-dimensional data where the number of covariates ( $p$ ) exceeds the sample size ( $n$ ). Although machine learning approaches have been proposed for competing risks, comprehensive comparative evaluations of these methods—specifically penalized regression, boosting, random forests, and deep learning—across diverse high-dimensional scenarios remain limited.

Methodology
The study conducts an extensive simulation-based comparison of four categories of competing risk methods:

Penalized Regression (PR): LASSO, SCAD, and MCP penalties applied to the Fine–Gray (PSDH) model.
Likelihood-based Boosting (CB): CoxBoost, a componentwise boosting approach for the PSDH model.
Random Forest (RF): Random Survival Forests adapted for competing risks using Gray's splitting rule.
Deep Learning (DH): DeepHit, a deep neural network framework for competing risks.

Experimental Design:

Data Generation: Simulated data was generated based on the Fine–Gray model with two competing events.
Scenarios: The study varied sample sizes ( $n = 200, 300, 500, 1000$ ), covariate dimensions ( $p = 24, 212, 512, 1012$ ), correlation structures (independent, exchangeable, AR(1)), sparsity levels, and predictor models (linear, quadratic, interaction). This resulted in 672 unique simulation scenarios with 10 replicates each.
Evaluation Metrics: Performance was assessed using:
- Variable Selection: True Positive Rate (TPR) and False Discovery Rate (FDR).
- Estimation Accuracy: Beta Error (sum of squared differences between true and estimated coefficients).
- Discrimination: Concordance Index ( $c$ -index) and Time-dependent Area Under the Curve ( $AUC_t$ ).
- Calibration: Integrated Brier Score ( $IBS_t$ ).
Real-World Application: The methods were applied to a high-dimensional melanoma gene expression dataset ( $n=214$ , $p \approx 47,000$ ) to illustrate practical utility.

Key Results

CoxBoost (CB): Demonstrated the most robust performance across high-dimensional settings ( $p \ge n$ ). It provided strong control of false discoveries, stable estimation accuracy, and competitive discriminative ability. CB maintained performance even as dimensionality increased and under correlated covariate structures.
Penalized Regression (PR): In low-to-moderate dimensional settings ( $n > p$ ), PR methods (particularly MCP and SCAD) showed improved calibration (lower IBS) compared to CB. However, in high-dimensional settings, PR methods exhibited inflated false discoveries and unstable estimation, with LASSO showing more stability but lower sensitivity than non-convex penalties.
Random Forest (RF): While capable of capturing non-linear effects, RF exhibited weaker overall performance in the current implementation. It identified broader variable sets with high false discovery rates and showed weaker discriminative ability compared to CB and PR.
Deep Learning (DeepHit): DH struggled with calibration accuracy in the implemented settings. While it can model complex non-linear relationships, it showed limited calibration performance and did not outperform CB or PR in the simulated high-dimensional regimes.
Computational Feasibility: CoxBoost scaled favorably with dimensionality (under 15 minutes for $n=1000, p=1012$ ). PR methods became computationally expensive in very high dimensions, while RF runtimes increased substantially with both $n$ and $p$ . DeepHit was the fastest but yielded non-competitive results in this specific configuration.

Significance and Claims
The paper claims to provide the first structured comparative evidence and preliminary guidelines for selecting competing risk models specifically in high-dimensional settings. The authors emphasize that their findings are based on specific implementations and tuning strategies (e.g., data generated from the Fine–Gray model, which structurally favors PR and CB).

The study concludes that CoxBoost offers a robust, well-balanced performance for variable selection, estimation stability, and discrimination, making it the recommended approach for high-dimensional competing risk data ( $p \gtrsim n$ ). Penalized Regression (specifically MCP/SCAD) is preferred when accurate probability calibration is the primary goal and dimensionality is moderate ( $n > p$ ). While Random Forest and Deep Learning are theoretically capable of handling non-linearities, their current implementations showed weaker performance in variable selection and calibration within the tested high-dimensional regimes.

The authors explicitly state that these results should be interpreted as a comparison within a widely used modeling paradigm (Fine–Gray) rather than a fully model-agnostic benchmark. They note that future research is needed to assess robustness under alternative data-generating mechanisms (e.g., cause-specific hazards or non-proportional effects) and to explore hybrid approaches or advanced tuning for deep learning models. The melanoma analysis serves as an illustrative use-case demonstrating method application rather than a source of new biological discovery.

Technical Summary: Comparative Review of Modern Competing Risk Methods in High-dimensional Settings

More like this