📊 statistics

When Are Scoring Rules Proper? Bridging Theory and Practice in Survival Model Evaluation

This paper demonstrates that while certain scoring rules for survival analysis are theoretically proper under ideal conditions, they become improperly biased toward underestimating survival in realistic scenarios with censoring or cure fractions, potentially leading to misleading model comparisons.

Original authors: John Zobolas, Raphael Sonabend, Riccardo De Bin, Johannes Piller, Philipp Kopper, Lukas Burk, Andreas Bender

Published 2026-07-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: John Zobolas, Raphael Sonabend, Riccardo De Bin, Johannes Piller, Philipp Kopper, Lukas Burk, Andreas Bender

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a doctor trying to predict how long a patient will live after a diagnosis. You don't just want to guess a single number, like "five years"; you want to paint a full picture of the future, saying, "There's a 90% chance they'll make it to year one, a 50% chance to year five, and so on." This is the world of survival analysis, a branch of statistics used in medicine, engineering, and finance to understand how long things last before they "break" or an event happens.

But here's the tricky part: in real life, you rarely get to see the full story. Some patients move away, some stop coming to appointments, or the study ends before everyone has passed away. In statistics, we call this censoring. It's like watching a movie but having to leave the theater before the credits roll; you know the plot up to a point, but the ending is a mystery. Because of this, judging how good a prediction model is becomes a game of "guessing the unseen." Scientists use special tools called scoring rules to grade these predictions. Think of these rules as a referee in a game: they check if the model's guesses match reality. A "proper" scoring rule is a fair referee that always gives the best score to the most honest and accurate model. If a referee is "improper," it might accidentally reward a liar or a bad guesser, leading us to trust the wrong model.

This paper, titled "When Are Scoring Rules Proper?", dives deep into the referee's rulebook for survival analysis. The authors, a team of statisticians and machine learning experts, investigate whether the most popular tools used to grade survival models are actually fair, especially when the "movie" is cut short by censoring. They focus on two main types of scoring rules: the Survival Brier Score (which is like measuring the distance between the predicted and actual outcome) and the Right-Censored Log-Loss (which punishes models for being surprised by the data).

The researchers found that the most popular tool, the Survival Brier Score (and its integrated version, the ISBS), has a hidden flaw. Imagine you are grading a student's prediction of a race finish time. If the race is stopped early (censoring) and you don't know who finished, the Brier Score acts like a grumpy teacher who assumes the students who didn't finish the race must have finished quickly. This causes the score to systematically bias models toward underestimating survival (predicting that people will die sooner than they actually might), even if those predictions are actually incorrect. The paper shows that this "improper" behavior gets worse the longer you wait to check the score and the more people drop out of the study. It's as if the referee is biased against the long-term winners, pushing the model to predict shorter lifespans to get a better score.

However, the paper offers a hero in the story: the Right-Censored Log-Loss (RCLL). This tool behaves like a much fairer referee. Even when the study ends early or some data is missing, it still correctly identifies the best model. The authors ran thousands of computer simulations to prove this. They found that while the Brier Score often gets confused and picks the wrong winner, especially in small groups of people or when many drop out, the Log-Loss stays steady and reliable.

There is a catch, though. The Log-Loss needs a bit more help to work perfectly: it requires the model to estimate the "density" of events (how likely an event is to happen at any exact second), which is a bit like needing a high-definition map instead of a blurry sketch. The paper shows that if you use a very detailed map (a dense grid of time points), the Log-Loss works beautifully. If the map is too blurry, the absolute score numbers might wiggle a bit, but the Log-Loss still correctly ranks which model is better than the other.

In short, the paper suggests that while the Survival Brier Score has been the star of the show for a long time, it might be time to switch referees. For the most honest and accurate model comparisons, especially in studies where people drop out or the study ends early, the Right-Censored Log-Loss is the safer, more trustworthy choice. The authors recommend using it with a detailed time grid and focusing on which model ranks highest rather than the exact score number, ensuring that in the race to find the best medical predictions, we aren't fooled by a biased referee.

Technical Summary: When Are Scoring Rules Proper? Bridging Theory and Practice in Survival Model Evaluation

1. Problem Statement

In survival analysis, model evaluation has traditionally relied on concordance indices, which assess only the ranking ability (discrimination) of predictions. However, modern survival models often generate probabilistic forecasts of the full time-to-event distribution, necessitating evaluation metrics that incorporate both calibration and discrimination. Scoring rules, such as the Brier score and Log-Loss, are standard for this purpose in classification and regression.

The central problem addressed in this paper is the properness of these scoring rules in the context of right-censored survival data. A scoring rule is "proper" if the expected loss is minimized when the predicted distribution matches the true data-generating distribution. While properness is well-understood for uncensored data, its behavior under censoring—particularly under independent censoring and in finite samples—is not fully characterized. Specifically, it is unclear whether commonly used metrics like the Survival Brier Score (SBS), Integrated Survival Brier Score (ISBS), and Right-Censored Log-Loss (RCLL) remain strictly proper when regularity conditions (such as vanishing tails) are violated, which is common in real-world studies with finite follow-up or cure fractions.

2. Methodology

The authors employ a dual approach combining theoretical derivation and extensive simulation experiments.

Theoretical Framework

Marginal Properness: The paper defines properness based on observable outcomes $(T, \Delta)$ , where $T = \min(Y, C)$ is the observed time and $\Delta$ is the event indicator. This "marginal" definition assumes independent censoring ( $Y \perp\!\!\perp C$ ) but allows predictions to depend on covariates. This contrasts with conditional properness definitions that condition on covariates $X$ .
Regularity Conditions: The analysis distinguishes between two settings:
1. Idealized: Survival and censoring distributions are absolutely continuous with vanishing tails ( $\lim_{t\to\infty} S(t) = 0$ ).
2. Practical: Distributions may have non-vanishing tails (residual mass) due to finite follow-up or cure fractions.
Loss Functions Analyzed:
- Exact Losses: Survival Continuous Ranked Probability Score (SCRPS), Negative Log-Likelihood (NLL), and Right-Censored Log-Loss (RCLL).
- IPCW Losses: Survival Brier Score (SBS), Integrated Survival Brier Score (ISBS), Integrated Binomial Log-Loss (IBLL), and Integrated Survival Absolute Score (ISAS). These utilize Inverse Probability of Censoring Weighting (IPCW) to adjust for censoring.

Simulation Experiments

Two primary simulation studies were conducted to validate theoretical findings and assess finite-sample behavior:

Empirical Properness Assessment: Using 10,000 simulations with Weibull-distributed event and censoring times, the authors evaluated the empirical loss difference between the true distribution and misspecified predictions across varying sample sizes ( $n=10$ to $10,000$). This tested whether misspecified models could achieve lower expected losses than the truth (violations of properness).
Sensitivity to Misspecification: Data was generated from a non-proportional hazards log-normal model with crossing survival curves. The study compared a correctly specified "Oracle" model against a sequence of increasingly misspecified models (e.g., omitted interactions, wrong parametric families, Cox models, Random Survival Forests). The ability of scoring rules to discriminate between these models was evaluated under low and high censoring rates.
Grid Resolution Sensitivity: For RCLL, which requires density estimation, the impact of the prediction grid resolution on absolute scores and relative rankings was tested.

3. Key Contributions and Results

Theoretical Findings

RCLL: Under independent censoring and standard regularity conditions (vanishing tails), the Right-Censored Log-Loss (RCLL) is strictly proper. However, if regularity conditions are violated (i.e., both event and censoring distributions have positive residual mass), RCLL can become improper, favoring predictions with smaller residual mass.
SBS and ISBS: The Survival Brier Score (SBS) and Integrated Survival Brier Score (ISBS) are strictly proper only if the residual tail mass is zero. When residual mass exists (common in finite follow-up), these scores become improper.
- Mechanism of Improperness: The expected SBS is minimized by a survival probability that systematically underestimates the true survival probability. The magnitude of this bias is driven by the residual tail mass ( $\epsilon$ ) and the ratio of the cumulative event probability to the censoring survival probability at the evaluation time.
- Time Dependence: The bias is negligible at early evaluation times but becomes pronounced at later times (e.g., the 90th percentile) and under heavy censoring.
Exact Losses: The SCRPS and NLL were shown to be improper under independent censoring, even under idealized conditions, due to the inability to properly account for the censoring mechanism without explicit weighting or density assumptions.

Empirical Findings

Finite-Sample Violations: In small samples ( $n < 50$ ), the SBS and ISBS frequently exhibit violations of properness, where misspecified models achieve lower losses than the truth. The frequency of these violations decreases as sample size increases and residual tail mass diminishes.
Discriminatory Power:
- SBS: Exhibits poor discrimination between models, especially at late evaluation times, due to the systematic bias introduced by residual mass.
- ISBS: More robust than SBS due to temporal integration, but still shows reduced discriminatory power and occasional improperness in small samples or under heavy censoring.
- RCLL: Demonstrated the most consistent behavior. It maintained strict properness in simulations and provided superior separation between correctly specified and misspecified models compared to ISBS.
Grid Resolution Effects: The absolute values of RCLL are sensitive to the density approximation grid. Coarser grids can artificially lower the RCLL for misspecified models. However, relative model rankings remain stable across different grid resolutions, provided all models are evaluated on the same grid.
Censoring Rate Impact: Absolute RCLL values are heavily influenced by the censoring rate (higher censoring shifts weight from density terms to survival terms). Therefore, comparing absolute RCLL values across datasets with different censoring rates is misleading; relative rankings within a dataset are the appropriate metric.

4. Significance and Recommendations

The paper demonstrates that theoretical improperness in survival scoring rules can translate into misleading model comparisons in practice. Specifically, the widely used ISBS and SBS may favor underestimating survival probabilities when follow-up is finite or cure fractions exist, potentially leading to the selection of suboptimal models.

Practical Guidelines Proposed:

Avoid Pointwise SBS: Do not use the pointwise Survival Brier Score, particularly at late evaluation times, as it is highly susceptible to improperness.
Use ISBS with Caution: If using ISBS, truncate the integration horizon (e.g., up to the 80th–90th percentile of follow-up) to mitigate tail instability and ensure test sample sizes are moderate ( $n \ge 50-100$ ).
Prefer RCLL: The Right-Censored Log-Loss is recommended as a theoretically grounded and empirically robust alternative. To use it effectively:
- Ensure predicted densities are approximated on a dense, consistent prediction grid across all compared models.
- Emphasize relative model rankings rather than absolute score values, especially in multi-dataset benchmarks or when censoring rates vary.
Sample Size: Reliable evaluation requires adequate test set sizes ( $n \ge 50-100$ ) to minimize finite-sample violations.

The authors conclude that while prior benchmarks relying on ISBS remain largely trustworthy under standard conditions, the field requires further methodological development to handle censoring and tail behavior more rigorously. The study underscores the necessity of complementing theoretical properness analysis with systematic empirical studies of discrimination and robustness.