📊 statistics

Position: Stop Chasing the C-index when Evaluating Survival Analysis Models

This position paper argues that the survival analysis community has overrelied on misaligned metrics like the C-index, urging researchers to adopt evaluation practices that explicitly account for censoring assumptions and align with specific modeling objectives to avoid misleading conclusions.

Original authors: Christian Marius Lillelund, Shi-ang Qi, Russell Greiner, Christian Fischer Pedersen

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Christian Marius Lillelund, Shi-ang Qi, Russell Greiner, Christian Fischer Pedersen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a coach trying to evaluate a new training program for a marathon. You have two goals:

Ranking: You want to know which runners are the fastest (who will finish first, second, third).
Timing: You want to know exactly when each runner will cross the finish line (e.g., 3 hours, 4 hours).

Now, imagine that during the race, some runners get injured or have to leave the track early. In statistics, this is called censoring. You don't know their final time, only that they didn't finish by the time they left.

This paper argues that the scientific community is currently making a huge mistake in how they evaluate survival models (the "training programs" for predicting when events happen, like disease progression or machine failure). They are using the wrong "stopwatches" and the wrong "scorecards."

Here is the breakdown of the paper's argument using simple analogies:

1. The Problem: The Wrong Scorecard

The paper claims that most researchers are obsessed with a metric called the C-index.

The Analogy: The C-index is like a judge who only cares about the order in which runners finish. If Runner A finishes before Runner B, the judge gives a high score.
The Flaw: The C-index doesn't care if Runner A finished in 2 hours or 100 hours, as long as they were faster than Runner B.
The Mismatch: Many researchers claim their model is great at predicting exact times (e.g., "This patient will get sick in 6 months"). But they only show the C-index score to prove it. It's like a chef claiming their soup is the perfect temperature, but the only proof they offer is that it tastes "saltier" than the other soup. The scorecard doesn't match the claim.

2. The Hidden Trap: The "Random" Assumption

The paper highlights a second, more dangerous problem: Censoring Assumptions.

The Analogy: Imagine you are judging the marathon, but some runners leave the track.
- Random Censoring: Runners leave because a bus picked them up at a random time, unrelated to how tired they are. (This is the "easy" scenario).
- Dependent Censoring: Runners leave because they are exhausted or injured. Their reason for leaving is directly tied to their performance. (This is the "hard" scenario).
The Mistake: Most researchers act as if all runners leave for random reasons (like the bus). They use standard formulas that assume this "randomness."
The Reality: In the real world, people often drop out of studies because they are getting sicker or machines break down faster. This is Dependent Censoring. When researchers use "Random" formulas on "Dependent" data, their results are like a map that says "North is South." The scores look good, but they are lying.

3. The "Double-Helix Ladder"

The authors introduce a concept called the Ladder Hypothesis.

The Analogy: Imagine a ladder where the rungs represent different levels of difficulty regarding how people drop out of a study.
- Bottom Rung: Easy dropouts (Random).
- Middle Rung: Medium dropouts (Independent).
- Top Rung: Hard dropouts (Dependent).
The Twist: The paper shows that the Models (the runners) have started climbing up the ladder to handle the hard dropouts. But the Metrics (the scorecards) are still stuck at the bottom rung.
The Result: You are trying to measure a climber who is halfway up a mountain using a ruler meant for the ground floor. The measurement is useless.

4. What They Found (The Evidence)

The authors looked at 92 recent papers (from 2023–2025) and found:

72% of them were "misaligned." They claimed to do one thing (like predict exact times) but used a scorecard that measured something else (like just ranking).
They ignored the "Dropout" rules. They rarely explained why they assumed people dropped out randomly, even when the data suggested otherwise.
The C-index is overused. It's the "default" choice, even when it's the wrong tool for the job.

5. The Solution: A New Checklist

The paper doesn't just complain; it offers a practical guide (a "Ladder") for researchers to fix this:

Know your goal: Are you trying to rank people, predict exact times, or check if probabilities are accurate?
Pick the right tool:
- If you want Ranking, use the C-index (but be careful!).
- If you want Exact Times, use error metrics (like MAE).
- If you want Probabilities, use Calibration metrics.
Check the "Dropout" rules: Be honest about why data is missing. If people drop out because they are sick (Dependent Censoring), you cannot use the standard "Random" scorecards. You need special tools that account for this bias.

Summary

The paper is a wake-up call: Stop chasing the C-index.

Just because a model gets a high score on the C-index doesn't mean it's good at predicting real-world outcomes, especially when people drop out of studies for reasons related to their health or the machine's condition. Researchers need to stop using a "ranking" scorecard for "timing" problems and stop pretending that dropouts are always random. If they don't, the scientific community is building a house of cards that looks impressive but collapses under real-world pressure.

Technical Summary: Stop Chasing the C-index when Evaluating Survival Analysis Models

Problem Statement

The current state of evaluation in survival analysis is characterized by a systematic misalignment between modeling objectives, evaluation metrics, and the underlying assumptions regarding censoring. Despite the defining characteristic of survival data being censoring (where event times are only partially observed), a substantial portion of recent literature relies on evaluation practices that ignore these structural features.

The authors identify two primary failures in current practice:

Objective-Metric Mismatch: Researchers frequently use discrimination-based metrics, particularly the C-index, to evaluate models even when the stated objective is accurate time-to-event estimation or probabilistic calibration. The C-index measures ranking ability but does not assess the numerical accuracy of predicted times or the calibration of survival probabilities.
Unjustified Censoring Assumptions: Many evaluations proceed as if censoring is "random" (independent of the event process), an assumption rarely stated, justified, or empirically corrected for. In reality, censoring may be independent (conditional on covariates) or dependent (related to the event process even after conditioning). Using metrics that assume random censoring on data with dependent censoring leads to biased performance estimates and misleading model comparisons.

A survey of 92 survival analysis papers published between 2023 and 2025 revealed that approximately 72% of studies used evaluation metrics that did not align with their stated objectives or failed to account for censoring assumptions, leading to potentially overstated performance claims.

Methodology and Framework

The paper proposes a structured framework for evaluating survival models, centered on the Ladder Hypothesis of Model-Metric Consistency. This hypothesis posits that a model and its evaluation metric must stand on the same "rung" of the assumption ladder. If a model relies on specific censoring assumptions (e.g., dependent censoring), the evaluation metric must also be valid under those same assumptions.

Key Conceptual Components

Censoring Assumptions: The authors formalize three distinct assumptions regarding the relationship between event time ( $E$ ) and censoring time ( $C$ ):
- Random Censoring: $E \perp\!\!\!\perp C$ . Censoring is unrelated to the event process.
- Independent Censoring: $E \perp\!\!\!\perp C \mid X$ . Censoring depends on covariates but is conditionally independent of the event time.
- Dependent Censoring: $E \not\perp\!\!\!\perp C \mid X$ . Censoring remains related to the event process even after conditioning on observed covariates.
Five Desiderata for Metrics: To guide metric selection, the authors propose five criteria:
- D1 (Proper Scoring Rule): The metric should be minimized/maximized only when predictions match the true distribution.
- D2 (Interpretability): Values should be in domain-relevant units (e.g., days, probabilities).
- D3 (Model-Agnostic): The metric should not depend on model internals or specific parameter choices.
- D4 (Sensitivity to Miscalibration): The metric must detect systematic over/under-estimation of probabilities.
- D5 (Robustness to Censoring): The metric must handle censoring consistent with the data's generation mechanism.
The Double-Helix Ladder: The authors visualize a gap between model development and metric development. While models have advanced to handle dependent censoring (e.g., via copulas), evaluation metrics have largely stagnated, remaining valid only under random or independent censoring assumptions. This creates a "twisted" ladder where advanced models are often evaluated with outdated, mismatched metrics.

Experimental Results

The authors conducted controlled experiments using synthetic data to demonstrate the consequences of model-metric mismatch.

Setup: A Cox Proportional Hazards (CoxPH) model was trained on data with a fixed event generation process. The censoring mechanism was varied across three scenarios: Random, Independent, and Dependent (induced via a Clayton copula with increasing Kendall's $\tau$ ).
Findings:
- Bias in Standard Metrics: As censoring deviated from the random assumption, standard metrics like Harrell's C-index and the Integrated Brier Score (IBS) became increasingly biased.
- Misleading Conclusions: In scenarios with dependent censoring, standard metrics often suggested model improvement or stability, even when the "oracle" performance (calculated using true event times) was degrading.
- Metric Failure: Metrics relying on marginal Inverse Probability of Censoring Weighting (IPCW) (e.g., Uno's C-index) failed to correct bias when censoring was dependent, as they assume independence given covariates or random censoring.

The results confirm that using a metric designed for a weaker assumption (e.g., random censoring) to evaluate a model or dataset with a stronger assumption (e.g., dependent censoring) yields unreliable performance estimates.

Key Contributions

Critical Review of Practices: The paper provides empirical evidence that the field is plagued by the overuse of the C-index and the neglect of censoring assumptions, with ~72% of recent papers exhibiting misalignment.
The Ladder Hypothesis: It introduces a guiding principle requiring explicit alignment between the statistical assumptions of the model, the data, and the evaluation metric.
Evaluation Guidelines: The authors provide a practical workflow (visualized in a checklist) for researchers to select appropriate metrics based on:
- The prediction target (e.g., risk ranking vs. time-to-event).
- The evaluation aspect (discrimination, error, calibration).
- The censoring assumption (random, independent, or dependent).
Identification of Gaps: The paper highlights that while methods for modeling dependent censoring exist, broadly accepted, unbiased evaluation metrics for such settings are currently lacking.

Significance and Claims

The authors argue that the widespread use of mismatched metrics leads to misleading model comparisons and prevents the community from accurately crediting methodological breakthroughs. They do not claim to have solved the problem of evaluating dependent censoring but rather call for a "recalibration" of the field.

The paper's primary significance lies in its call to action: researchers must explicitly state their censoring assumptions and choose metrics that are valid under those assumptions. Until robust, assumption-aligned metrics for dependent censoring are developed and validated, performance claims for state-of-the-art models in such settings should be viewed as speculative. The authors advocate that rigorous, assumption-aware evaluation is the necessary next frontier for progress in survival analysis.