📊 statistics

Assessing survival models by interval testing with Poisson-binomial distributions

This paper proposes a novel model selection technique for parametric survival models that utilizes Poisson-binomial distributions to evaluate goodness-of-fit across specific time intervals, offering a more granular assessment of model incompatibility than traditional relative metrics like AIC or BIC while maintaining controlled Type I error rates.

Original authors: Ben Lee

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ben Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a doctor trying to predict how long a patient might live after a new treatment. You have data from a clinical trial, but the trial ended before everyone passed away (this is called "right-censored" data). To make predictions about the future, you fit a mathematical curve (a "survival model") to the data.

The problem is: How do you know if your curve is actually a good fit?

Currently, scientists mostly use tools like AIC and BIC. Think of these like a "popularity contest." They tell you which curve is better than the others, but they don't tell you if any of the curves are actually terrible. They also don't tell you where the curve goes wrong. Is it wrong at the beginning? Is it wildly inaccurate at the end?

Ben Lee's paper proposes a new way to check these curves, acting like a spot-check inspector rather than just a popularity judge.

The Core Idea: The "Time Interval" Check

Instead of looking at the whole curve at once, the author suggests chopping the timeline into specific chunks (intervals) and counting the events (deaths) in each chunk.

The Analogy: The Bus Stop
Imagine you are trying to predict how many people will get off a bus at various stops.

The Model: You have a computer program that predicts, "At Stop 1, 5 people will get off. At Stop 2, 3 people will get off."
The Reality: You watch the bus. At Stop 1, 20 people get off. At Stop 2, 0 people get off.
The Old Way (AIC/BIC): These tools might say, "Well, your prediction for Stop 2 was better than the other guy's prediction, so you win." But they ignore the fact that you were completely wrong about Stop 1.
The New Way (This Paper): This method looks at Stop 1 and says, "Hey! You predicted 5, but 20 got off. That's a huge mismatch!" It flags that specific stop as a problem.

How the Math Works (Simply)

The paper uses a statistical tool called the Poisson-binomial distribution.

The "Poisson" part: This helps handle the fact that we don't know exactly when people will leave the study (censoring). It's like knowing how many people might get off the bus, even if some leave early for other reasons.
The "Binomial" part: This calculates the odds. If the model says there is a 10% chance of an event in a specific time window, and 100 people are at risk, we expect 10 events. If we see 50, the math tells us, "That is extremely unlikely to happen by chance."

The author suggests two ways to slice the time:

Censor-Defined Intervals: Cut the time every time someone leaves the study (censors).
Evenly-Spaced Intervals: Cut the time into 10 equal chunks (e.g., every 5 months).

The "Spotlight" Tests

Once the time is chopped up, the author uses two "flashlights" to find the bad spots:

The "Bonferroni" Flashlight: This is a strict inspector. It looks for any single time chunk where the model is so wrong that it would be a statistical miracle to happen by chance. If it finds one, it screams, "This model is broken!"
The "Fisher" and "PAVSI" Flashlights: These look at the whole picture. They combine the results of all the time chunks into one big score.
- Fisher (TFT): Sensitive to extreme errors. If the model is wildly wrong in one place, this score goes up.
- PAVSI: Counts how many chunks are "suspicious." It's like a teacher counting how many questions a student got wrong on a test, rather than just looking at the final grade.

What the Paper Found

The author ran thousands of computer simulations (like a video game where they created fake patient data) to see if this new method works.

It doesn't cry wolf: The method is very good at not rejecting a good model. If the model is actually correct, the test rarely says it's wrong (the "Type I error" stays low).
The "Evenly-Spaced" approach is better: When they chopped the time into 10 equal pieces, the test worked well. When they chopped it based on when people left the study, the test became too "conservative" (it was too scared to say a model was bad, even when it was).
Real-world examples:
- In one example, a simple model looked okay visually, but this new method found a massive error at the very end of the timeline. This is crucial because in medicine, predicting the long-term future is often the most important part.
- In another example, they tested 7 different models. The old methods (AIC/BIC) couldn't tell them which was best. The new method showed that 6 out of 7 models had specific time periods where they failed completely.

The Bottom Line

This paper introduces a new tool for checking survival models. Instead of just asking, "Which curve is the least bad?", it asks, "Where exactly does this curve fail, and is it failing so badly that we should throw it out?"

It uses a clever statistical trick (Poisson-binomial) to handle messy data and provides a "map" of errors, showing researchers exactly which time periods their predictions are unreliable. The simulations prove that this map is accurate and doesn't give false alarms.

Technical Summary: Assessing Survival Models by Interval Testing with Poisson-binomial Distributions

Problem Statement
In regulatory submissions for pharmaceutical products, particularly within health technology assessment (HTA), selecting the appropriate parametric survival model is critical for economic evaluations and extrapolating treatment efficacy beyond trial durations. While the number of available modeling approaches (including standard parametric, flexible spline-based, and external data-integrated models) is increasing, model selection remains challenging. Current standard practices rely heavily on Akaike Information Criterion (AIC), Bayesian Information Criterion (BIC), and subjective expert opinion. The paper argues that these methods primarily offer relative goodness-of-fit comparisons, failing to provide insight into whether all considered models are incompatible with observed data. Furthermore, they do not identify specific time intervals where a fitted model performs poorly, which is vital for assessing the validity of long-term extrapolations.

Methodology
The authors propose a goodness-of-fit testing framework based on grouping survival data into time intervals and evaluating the observed event counts against expected counts derived from a fitted model.

Interval Construction:
- Censor-Defined Intervals: Intervals are defined by the unique times of right-censored observations ( $t'_{j,0}$ ). This approach, rooted in Akritas (1988), creates subintervals $(t'_{j-1,0}, t'_{j,0}]$ .
- Specified Intervals: The authors extend the method to allow for user-defined intervals (e.g., 10 evenly-spaced intervals up to the last censor). These specified intervals ( $V_k$ ) are constructed by merging the censor times with the user-defined boundaries, creating a set of subintervals ( $I'_i$ ).
Statistical Distribution:
- Under the null hypothesis that the model $M(\beta)$ correctly describes the event generating process, the number of events in any interval is modeled using a Poisson-binomial distribution.
- For a specific interval, the number of events follows a binomial distribution $Binomial(N_{risk}, p)$ , where $N_{risk}$ is the number of patients at risk and $p$ is the probability of an event occurring in that interval given survival to the start of the interval.
- When aggregating subintervals within a specified interval, the sum of independent binomials with varying probabilities follows a Poisson-binomial distribution. This approach avoids the asymptotic normality assumptions required by traditional chi-squared tests (e.g., Akritas).
Test Statistics:
- Individual Interval Testing: P-values are calculated for each interval. The authors utilize "midpoint p-values" for reproducibility, though they acknowledge "randomized p-values" for theoretical uniformity.
- Bonferroni Test: Used to strongly control the familywise error rate. An interval is flagged if its p-value is extreme (very low or very high) relative to the number of intervals.
- Transformed Fisher Test (TFT): A continuous test statistic ( $T_{cont} = -2 \sum \log(U_j)$ ) that aggregates p-values to detect an unusual concentration of extreme values (both high and low), following a $\chi^2$ distribution.
- Protection Against Very Small Intervals (PAVSI): A discrete test statistic ( $T_{pavsi}$ ) that counts the number of intervals with p-values falling outside the central 95% (i.e., $p < 0.025$ or $p > 0.975$ ). This statistic follows a binomial distribution and is designed to mitigate the influence of single, extremely extreme p-values.

Key Contributions and Results

Identification of Local Misfit: Unlike AIC/BIC, the proposed method highlights specific time intervals where models fail, providing actionable insights for model selection and extrapolation validity.
Simulation Validation: A simulation study involving 12 scenarios (varying patient numbers and data maturity) and 10,000 iterations per scenario was conducted.
- Type I Error Control: For randomized p-values, both the censor-defined and 10 evenly-spaced interval approaches maintained empirical Type I error rates close to the nominal 5% level.
- Midpoint P-values: When using midpoint p-values (recommended for practical application), the censor-defined interval approach proved highly conservative (Type I error often near 0 or <2%), particularly for the TFT statistic in immature data. In contrast, the 10 evenly-spaced interval approach performed better, with Type I error rates generally ranging between 0.02 and 0.05, approaching the nominal level as patient numbers increased.
Application Examples:
- Example 1 (Exponential Model): Demonstrated the ability to detect poor fit at the tail of the curve (extrapolation region) via Bonferroni rejection, which visual inspection alone might miss.
- Example 2 (HTA Scenario): Compared seven parametric models. The censor-defined approach failed to detect poor fit due to small interval sizes, whereas the 10 evenly-spaced approach identified significant misfit in 6 of 7 models, guiding the selection of more flexible models (e.g., splines).

Significance and Claims
The paper claims that the proposed Poisson-binomial interval testing method offers a significant advancement over classical approaches by:

Removing Asymptotic Assumptions: It does not rely on the asymptotic normality assumptions inherent in chi-squared tests, making it suitable for smaller sample sizes or complex censoring patterns.
Granular Diagnostic Capability: It moves beyond relative model ranking to provide absolute goodness-of-fit assessments, specifically identifying where in the time horizon a model is incompatible with data.
Practical Utility: The authors recommend the 10 evenly-spaced interval approach combined with midpoint p-values for practical use, as it balances Type I error control with the ability to detect misfit in real-world scenarios (such as the HTA example provided).

The authors conclude that while the method is a robust tool for initial model overview and rejection, it should be used in conjunction with other metrics (like AIC/BIC) and visual inspection, rather than as a sole decision criterion. They note that the method is flexible enough to accommodate various interval selection strategies and could be extended to risk-score models.