📊 statistics

Efficient Cumulative Incidence Estimation in Biobank Studies Using All Prevalent and Incident Events

This paper proposes a novel cumulative incidence function estimator for biobank studies that improves upon existing methods by incorporating both prevalent and incident disease cases, thereby enabling accurate incidence estimation for diseases with early onset and long survival, as demonstrated through theoretical analysis, simulations, and an application to UK Biobank cancer data.

Original authors: David M. Zucker, Malka Gorfine

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: David M. Zucker, Malka Gorfine

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Counting Illness in a "Snapshot" World

Imagine you are trying to count how many people in a city get a specific disease (like breast cancer) over their entire lives. In the old days, researchers would pick a group of healthy people, wait for them to get sick, and watch what happened. This is like starting a race at the starting line and watching everyone run.

But modern "Biobanks" (huge databases of health records) work differently. They don't start a race; they take a snapshot of a crowd that has already been walking for a long time.

The Setup: They only let people join the study if they are between certain ages (say, 40 and 69).
The Problem: By the time these people join, some have already gotten sick (these are called prevalent cases). Some get sick after they join (these are incident cases). Some get sick and die quickly. Some get sick and live for decades. Some never get sick at all.

The paper's authors, Zucker and Gorfine, are trying to build a better "mathematical camera" to estimate the true risk of getting this disease, using all the messy data from these snapshots.

The Three Old Cameras (and why they were blurry)

To understand their new invention, we first need to see why the old ways of looking at the data were flawed.

1. The "Aalen-Johansen" Camera (The Strict Gatekeeper)

How it works: This method only counts people who get sick after they enter the study. It completely ignores anyone who was already sick when they walked through the door.
The Flaw: It's like a teacher grading a test but refusing to count the answers of any student who arrived late, even if they had the right answers.
The Result: It misses a huge chunk of the data (the "prevalent" cases). It also can't tell you the risk of getting sick at age 30, because no one in the study was 30 when they started. It throws away valuable information.

2. The "GZS" Camera (The "Death-Only" Filter)

How it works: This method tries to include the people who were already sick when they joined. However, it has a strict rule: it only counts a sick person if they died during the study period. If a person got sick and is still alive at the end of the study, this camera ignores them.
The Flaw: This works great for diseases that kill quickly (like a fast-moving flu). But for diseases like breast cancer, where people often live for 20 or 30 years after diagnosis, this camera is blind.
The Analogy: Imagine trying to count how many people bought a specific car by only looking at people who crashed their cars. If most people keep their cars for 20 years without crashing, you will vastly underestimate how many people bought that car.
The Result: In the UK Biobank data (which includes breast cancer), this method significantly underestimated the risk because it ignored all the survivors.

3. The "Proposed" Camera (The New, All-Seeing Lens)

How it works: The authors built a new mathematical tool that counts everyone.
- It counts people who were sick before joining.
- It counts people who got sick after joining.
- It counts people who died after getting sick.
- Crucially: It counts people who got sick and are still alive at the end of the study.
The Magic: It uses a clever weighting system to balance the data. It realizes that if you see a healthy 40-year-old, they are "more likely" to have survived to 40 than a sick person who died at 30. It adjusts the math to account for these differences without throwing anyone away.

Why This Matters: The "Breast Cancer" Example

The paper tested this new camera on real data from the UK Biobank, specifically looking at breast cancer.

The Situation: Breast cancer often happens early, and women often live for a long time after diagnosis.
The Old Results:
- The "Strict Gatekeeper" (Aalen-Johansen) gave a reasonable answer but had very wide "margins of error" (like a blurry photo) because it ignored half the sick people.
- The "Death-Only" filter (GZS) gave a very low, incorrect answer because it ignored all the women who were still alive.
The New Result:
- The new method gave an answer very similar to the "Strict Gatekeeper" (meaning it was accurate).
- But, because it used all the data (including the survivors and the people already sick), the "margin of error" was half as wide.
- The Analogy: It's like taking a photo with a high-resolution lens instead of a blurry one. You see the same scene, but the details are much sharper, and you are much more confident in what you are seeing.

The Bottom Line

The authors created a new statistical tool that fixes a major blind spot in medical research.

It uses everything: It doesn't throw away people who were already sick or people who are still alive.
It's more accurate: It works well even for diseases where people live a long time after diagnosis (like many cancers).
It's more efficient: It gives a clearer, more precise picture of disease risk using the same amount of data, effectively doubling the "power" of the study.

In short, they found a way to stop throwing away half the puzzle pieces, allowing scientists to see the full picture of disease risk much more clearly.

Technical Summary: Efficient Cumulative Incidence Estimation in Biobank Studies

Problem Statement
The paper addresses the challenge of estimating the Cumulative Incidence Function (CIF) for a specific disease using population-based biobank data. Biobank studies typically recruit subjects within a specific age window $[R_L, R_U]$ (e.g., ages 40–69 in the UK Biobank) and follow them prospectively. This design introduces two critical statistical complexities:

Left Truncation: Individuals are only observed if they survive to their recruitment age $R_i$ . Consequently, incidence must be estimated conditional on survival to $R_L$ , denoted as $G_1(t | T_2 > R_L)$ .
Prevalent vs. Incident Cases: The data includes "prevalent cases" (individuals diagnosed before recruitment) and "incident cases" (diagnosed after recruitment). Furthermore, the disease is non-terminal; individuals can die after diagnosis or remain alive at the end of follow-up.

Existing methods fail to efficiently handle the combination of these factors. The standard left-truncation adjusted Aalen-Johansen (AJ) estimator excludes prevalent cases, effectively estimating a different target: the CIF conditional on being disease-free at recruitment ( $G_1(t | T_1 > R_L, T_2 > R_L)$ ). Inverse Probability Weighting (IPW) methods, such as those by Chang and Tzeng (2006), Vakulenko-Lagun et al. (2017), and Gorfine et al. (2025) (GZS), attempt to include prevalent cases but suffer from significant limitations. Specifically, the GZS estimator assigns positive mass only to disease cases where the individual died during the follow-up period. This leads to substantial bias and inefficiency in scenarios where survival after disease onset is long (a common feature for diseases like breast cancer), as many prevalent and incident cases remain alive at the end of the study and are thus discarded by the GZS method.

Methodology
The authors propose a novel, fully nonparametric CIF estimator, denoted $\hat{G}_1(t^*_1)$ , designed to incorporate all observed disease cases (both prevalent and incident) regardless of whether the individual died during the study or survived to the end of follow-up.

The estimator is constructed within the illness-death model (semi-competing risks), where transitions occur from healthy (0) to diseased (1) and from diseased (1) to dead (2), or directly from healthy (0) to dead (2). The core of the proposed method involves a weighting scheme derived from the conditional joint density of the data. The estimator takes the form:
$\hat{G}_1(t^*_1) = \frac{1}{n} \sum_{i=1}^n \delta_{1i} I(V_{1i} \le t^*_1) \hat{Q}(V_{1i})$
where $\delta_{1i}$ indicates disease onset, $V_{1i}$ is the observed time of the first event (diagnosis or death), and $\hat{Q}$ is a data-driven weight function.

To ensure consistency, $\hat{Q}$ is chosen as the inverse of an estimated function $\hat{B}(V_{1i})$ , which accounts for:

The distribution of recruitment ages conditional on survival ( $f_{R|R \le T_2}$ ).
The survival function of death ( $S_{T_2}$ ).
The conditional survival of death given disease onset ( $S_{T_2|T_1}$ ).
The distribution of the time from recruitment to censoring ( $S_W$ ).

The conditional survival function $S_{T_2|T_1}$ is estimated using a kernel-based nonparametric regression approach (Nelson-Aalen version of Beran's estimator). The bandwidth for the kernel is selected via a $K$ -fold cross-validation procedure based on martingale residuals.

Key Contributions

Novel Estimator: The paper introduces the first fully nonparametric and consistent CIF estimator in the illness-death model that utilizes all disease cases, including prevalent cases and those who survive the study period.
Theoretical Properties: The authors establish the consistency and asymptotic normality of the proposed estimator. They provide explicit formulas for the influence function, enabling the construction of pointwise confidence intervals and simultaneous confidence bands.
Efficiency Gains: By incorporating data that previous methods discard (specifically, prevalent cases and long-term survivors), the new method achieves substantially higher efficiency, resulting in narrower confidence intervals.
Software Implementation: An R package, illdthCIF1, is provided to implement the proposed estimator.

Results

Simulation Study: The authors conducted extensive simulations across various scenarios involving different distributions for disease onset, death, and censoring.
- In scenarios with short follow-up relative to post-diagnosis survival (where many patients remain alive), the GZS estimator exhibited serious bias, underestimating the CIF. The proposed estimator remained largely unbiased, with only slight bias in specific configurations that decreased with sample size.
- In scenarios with long follow-up, the GZS estimator was unbiased, but the proposed estimator demonstrated lower standard deviation (higher efficiency).
- The proposed method provided satisfactory coverage for both pointwise confidence intervals and simultaneous confidence bands in most scenarios.
UK Biobank Application: The method was applied to breast and other cancer data from the UK Biobank (approx. 250,000 female participants).
- For breast cancer, where post-diagnosis survival is often long, the GZS estimator systematically underestimated the CIF compared to the AJ and proposed methods.
- The proposed estimator and the AJ estimator produced similar point estimates for breast cancer, but the proposed method yielded significantly narrower confidence bands. Specifically, the average width of the 95% simultaneous confidence band for breast cancer was reduced from 0.0068 (AJ) to 0.0035 (proposed), representing a nearly two-fold efficiency gain.
- For cancers with shorter post-diagnosis survival (e.g., lung, leukemia), all three methods produced similar estimates, as deaths were observed more completely.

Significance
The paper claims that its primary significance lies in bridging a methodological gap in biobank analysis. While biobanks offer unprecedented power for studying rare diseases and long-latency endpoints, standard estimators either discard valuable prevalent data (AJ) or fail to account for long survival after diagnosis (GZS). The proposed method allows for the efficient and consistent estimation of disease incidence conditional on recruitment survival, utilizing the full spectrum of available data. This is particularly crucial for diseases like breast cancer, where ignoring prevalent cases or long-term survivors leads to biased or inefficient estimates. The authors emphasize that their approach aligns estimation with the true conditional estimand of biobank studies while maximizing statistical efficiency.