📊 statistics

Multi-state Models For Disease Histories Based On Longitudinal Data

This paper extends piecewise exponential additive models (PAMs) to analyze multi-state disease histories with complex longitudinal data challenges like dependent left-truncation and interval-censoring, demonstrating their effectiveness through simulations and a UK Biobank application that reveals distinct genetic and age-related risk factors for chronic kidney disease onset versus progression.

Original authors: Simon Wiegrebe, Johannes Piller, Mathias Gorski, Merle Behr, Helmut Küchenhoff, Iris M. Heid, Andreas Bender

Published 2026-03-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Simon Wiegrebe, Johannes Piller, Mathias Gorski, Merle Behr, Helmut Küchenhoff, Iris M. Heid, Andreas Bender

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand the life story of a disease, not just as a single event, but as a journey with many stops, detours, and possible endings. This paper is about building a better map for that journey.

Here is the story of the paper, broken down into simple concepts with some creative analogies.

1. The Problem: The "Journey" is Complicated

Think of a disease like Chronic Kidney Disease (CKD) as a traveler moving through a series of train stations:

Station 0: Healthy.
Station 1: Mild Kidney Trouble.
Station 2: Severe Kidney Trouble.
Station 3: End-Stage Kidney Disease (The final stop).
Station 4: Death (Another final stop).

Scientists want to know: If a person is at Station 1, what are the odds they will reach Station 2 next year? What if they are 60? What if they have diabetes?

The problem is that real-world data is messy.

The "Late Arrival" Problem (Dependent Left-Truncation): Imagine you only start watching the train passengers when they arrive at Station 1. You don't see them at Station 0. If you only look at people who made it to Station 1, you might miss the fact that the people who got there young are different from the people who got there old.
The "Different Clocks" Problem (Multiple Time Scales): Is the risk of moving to the next station based on how old the person is (their biological clock), or how long they have been sick (their disease clock)? A 50-year-old who just got sick is different from a 50-year-old who has been sick for 20 years.
The "Blind Spot" Problem (Interval Censoring): Doctors don't check patients every day. They check them every 6 months. If a patient gets worse between check-ups, we only know it happened sometime in that 6-month window, not exactly when. It's like watching a movie where 10 minutes are missing every hour.
The "Fake Connection" Problem (Index Event Bias): This is the trickiest part. Imagine two independent causes of a car crash: bad brakes and bad tires. If you only study cars that already crashed (the "index event"), you might falsely conclude that "bad brakes" and "bad tires" are opposites. Why? Because if a car had bad brakes, it didn't need bad tires to crash. But if you only look at crashed cars, you see a weird negative link between the two. In medicine, this can make a genetic gene look like it protects you from a disease, when it actually doesn't.

2. The Solution: A "Smart, Flexible Map" (PAMs)

The authors propose a new tool called Piecewise Exponential Additive Models (PAMs).

Think of traditional statistical models as a rigid ruler. They try to force the data into a straight line or a perfect curve. If the data bends weirdly, the ruler breaks or gives a wrong answer.

The PAM is like a flexible, bendable ruler made of smart rubber.

It can stretch and shrink to fit the actual shape of the data without forcing it into a pre-set box.
It can handle the "missing movie scenes" (interval censoring) by looking at the whole chunk of time rather than guessing a single exact moment.
It can juggle multiple clocks (age vs. time since sickness) at the same time.

3. The Experiment: Testing the Map

The authors built a computer simulation (a "virtual world") to test their new map.

They created fake patients with fake disease histories.
They introduced all the messy problems mentioned above (late arrivals, missing dates, fake connections).
The Result: Their flexible rubber ruler (PAM) handled the mess much better than the old rigid rulers. It could still find the true risks even when the data was incomplete or tricky.
Key Finding: They found that sometimes, using just one "clock" (like just age) is actually more robust and less confusing than trying to juggle too many clocks at once, unless you really need the extra detail.

4. The Real World Test: The UK Biobank

They took their new map and applied it to a massive real-world dataset from the UK Biobank (over 140,000 people). They tracked the journey of Chronic Kidney Disease.

What did they discover?

The "Early Starter" Risk: If you get mild kidney trouble at a young age (say, 40), your risk of it getting worse is much higher than if you get it at 70. It's like starting a fire in a dry forest vs. a wet one; the early fire spreads faster.
The Genetic Mystery (The "Index Event" Fix): There is a famous gene (rs77924615) known to affect kidneys.
- The Naive View: At first glance, it looked like this gene was "protective" against the disease getting worse (moving from Mild to Severe). It seemed like a superhero gene!
- The Real View: The authors realized this was a "Fake Connection" (Index Event Bias). The gene makes you more likely to get sick early. Because you got sick early, you were already in the "sick group" when the study started. When they adjusted for other factors (like diabetes and BMI) that were hidden in the mix, the "superhero" effect vanished. The gene makes you get sick, but it doesn't actually stop the disease from getting worse once you have it.

The Bottom Line

This paper gives doctors and researchers a better, more flexible way to track diseases that happen in stages. It teaches us to be careful about "fake connections" in data and shows that sometimes, the simplest way to measure time (just age) is better than over-complicating it, as long as you have a flexible tool to handle the messy, missing pieces of the puzzle.

In short: They built a smarter GPS for disease progression that doesn't get lost when the road is bumpy, the map is torn, or the traffic is confusing.

1. Problem Statement

The paper addresses the statistical challenges inherent in analyzing multi-stage disease histories (e.g., Chronic Kidney Disease progression from Healthy $\to$ Mild $\to$ Severe $\to$ End-Stage) derived from longitudinal data such as registries and biobanks. While multi-state models are ideal for studying transitions between disease stages in the presence of competing risks, their application is complicated by four specific statistical hurdles:

Dependent Left-Truncation: Subjects enter the risk set for a specific transition (e.g., progression from Mild to Severe CKD) at different times (state-entry times). Standard methods assume these entry times are independent of event times, but in reality, early disease onset may correlate with higher progression risk, violating this assumption.
Multiple Time Scales: Disease progression is often best described by multiple time scales simultaneously (e.g., chronological age, time since disease onset, and time since progression). Traditional models (like Aalen-Johansen or standard Cox models) typically rely on a single time scale, potentially missing critical dynamics.
Index Event Bias (IEB): Conditioning on an "index event" (e.g., restricting analysis to patients who already have Mild CKD) can induce spurious negative correlations between independent risk factors, leading to biased effect estimates (attenuation or reversal) if confounders are omitted.
Interval-Censoring: Longitudinal data often lacks exact event times; events are only known to have occurred between two observation points (intervals). Standard survival methods often struggle with this, particularly when combined with the other challenges.

2. Methodology

The authors propose and evaluate a Piecewise Exponential Additive Model (PAM) framework adapted for multi-state modeling.

Core Framework: The method transforms survival data into a Piecewise Exponential Data (PED) format, effectively converting the survival task into a Poisson regression task.
- Baseline Hazards: Instead of assuming a parametric distribution (like Weibull) or using non-parametric step functions, PAMs estimate baseline hazards as smooth functions over time using penalized splines. This allows for flexible, non-linear modeling without strict distributional assumptions.
- Multi-State Extension: The framework handles transitions between states by creating transition-specific datasets where competing risks are treated as censored events.
Addressing Specific Challenges:
- Multiple Time Scales: The authors define two model types:
  - Stratified Single Time Scale (SSTS): Uses one time scale (e.g., age) with stratification.
  - Multiple Time Scales (MTS): Explicitly models hazards as functions of multiple time scales (e.g., age + time since onset) simultaneously. They introduce "helper variables" to ensure model identifiability when estimating effects across different transitions.
- Dependent Left-Truncation: The framework incorporates state-entry times (e.g., age at CKD onset) directly as covariates (via smooth splines) to adjust for the dependence between entry time and event time.
- Interval-Censoring: The model uses interval mid-points (or end-points) as estimation points. The Poisson likelihood equivalence allows for flexible handling of these intervals.
- Index Event Bias: The authors demonstrate that including all relevant risk factors (confounders) in the model or using propensity score weighting can correct the bias induced by conditioning on the index event.

3. Key Contributions

Framework Extension: Successfully extends PAMs to complex multi-state settings, enabling the simultaneous handling of dependent left-truncation, multiple time scales, and interval-censoring.
Identifiability Solutions: Proposes specific strategies (using ordered factor variables or factor smooths) to resolve rank-deficiency issues when estimating multiple time scales and state-entry times in a single model.
Comprehensive Simulation Studies: Conducts extensive simulations to evaluate the performance of SSTS vs. MTS models under various Data Generating Processes (DGPs), specifically testing:
- Recovery of baseline hazards and fixed effects under model misspecification.
- Quantification and correction of Index Event Bias.
- Impact of interval-censoring on estimation accuracy.
Real-World Application: Applies the framework to a large-scale UK Biobank dataset ( $n=142,667$ ) to model CKD onset and progression, providing detailed transition probabilities and risk factor associations.

4. Key Results

Simulation Findings

Time Scales: The SSTS PAM (single time scale) proved more robust to model misspecification regarding baseline hazard estimation than the MTS PAM. While MTS models performed well when correctly specified, they suffered significant coverage drops (down to 69%) when misspecified. However, for estimating fixed effects (risk factor coefficients), both approaches were robust.
Interval-Censoring: PAMs using interval mid-points recovered fixed effect hazards with near-nominal coverage and negligible bias, even under heavy censoring. However, baseline hazard recovery was suboptimal (50–75% coverage) compared to exact time estimation, though still superior to parametric AFT models when the true distribution was unknown.
Index Event Bias: Simulations confirmed that omitting confounders in a subpopulation restricted by an index event induces negative correlation between risk factors and biases effect estimates toward zero (attenuation) or reverses them. Including the omitted confounders or using propensity weighting successfully removed this bias.

UK Biobank Application (CKD)

Disease Dynamics: The study confirmed that CKD progression risk is highest for individuals with early onset and increases with age.
- For a subject with Mild CKD onset at age 50 and progression to Severe CKD at age 60, the probability of reaching End-Stage Kidney Disease (ESKD) rises from 15% at age 61 to $>60\%$ over the next decade.
Genetic Variant (rs77924615):
- Naive Model: Initially appeared to have a protective effect on CKD progression (hazard ratio < 1).
- Adjusted Model: After adjusting for confounders (diabetes, BMI, uACR, etc.), the protective effect vanished (coefficient became non-significant and slightly positive).
- Conclusion: The initial "protective" finding was an artifact of Index Event Bias. The variant is associated with CKD onset but has no significant effect on progression once other risk factors are accounted for.
Model Selection: The SSTS PAM provided a better fit (lower AIC) than the MTS PAM for the CKD data, reinforcing the simulation finding that single time-scale models are often sufficient and more robust.

5. Significance

This paper provides a robust, flexible statistical toolkit for modern epidemiological research utilizing large-scale longitudinal data.

Methodological Advancement: It moves beyond the restrictive Markov assumption and single time-scale limitations of traditional multi-state models, allowing researchers to incorporate history-dependent information (like age at onset) directly into the hazard function.
Bias Correction: It offers a clear, practical demonstration of how Index Event Bias can distort genetic and clinical risk factor associations in disease progression studies and how to correct it.
Practical Utility: By leveraging the pammtools R package and mgcv, the authors provide a computationally efficient solution (using fast REML) for analyzing massive datasets (like the UK Biobank) that were previously difficult to model with complex multi-state structures.
Clinical Insight: The application to CKD highlights the importance of distinguishing between disease onset and progression risk factors, preventing erroneous conclusions about genetic variants that could misguide therapeutic targets.

In summary, the work establishes Piecewise Exponential Additive Models as a superior approach for analyzing complex disease trajectories, effectively balancing flexibility, computational efficiency, and robustness against common longitudinal data pitfalls.