📊 statistics

Semi-supervised Method for Risk Prediction with Doubly Censored EHR Data

This paper proposes a novel semi-supervised learning framework that effectively integrates limited gold-standard labels with abundant surrogate outcomes to improve risk prediction accuracy under the challenging conditions of doubly censored electronic health record data.

Original authors: Jie Zhou, Enhao Wang, Xuan Wang

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jie Zhou, Enhao Wang, Xuan Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: When do people actually get Type 2 Diabetes?

You have a massive library of patient records (Electronic Health Records, or EHRs) containing millions of people. However, you face two major problems that make solving the mystery difficult:

The "Double Blind" Problem: You don't always know the exact date the disease started.
- Sometimes, a patient walks into the hospital already sick. You know they were sick before they arrived, but you don't know when (this is Left Censoring).
- Sometimes, a patient leaves the hospital system while still healthy. You know they were healthy up to that point, but you don't know if they got sick the next day or ten years later (this is Right Censoring).
- This "double blind" situation makes it hard to calculate the true risk.
The "Gold Standard" vs. "Clue" Problem:
- The Gold Standard (Labeled Data): To know the exact truth, a team of experts has to manually read through thousands of old paper charts. This is incredibly slow, expensive, and tiring. Because of this, you only have the "true answer" for a tiny handful of patients (say, 1,600 people).
- The Clues (Unlabeled Data): For the other 113,000+ patients, you don't have the manual review. Instead, you have "surrogate clues," like the date a specific code (e.g., "Diabetes") first appeared in their computer file. These clues are easy to get for everyone, but they are often wrong or imprecise (maybe the code was a mistake, or maybe it was entered late).

The Old Way vs. The New Way

The Old Way (Supervised Learning):
Previously, researchers would only look at the tiny group of 1,600 patients with the "Gold Standard" answers. They would ignore the 113,000 others because their data was "noisy" or "wrong." This is like trying to solve a murder mystery by only interviewing the one witness who saw the crime, while ignoring 100 other people who saw blurry shadows or heard noises. Your conclusion would be shaky and imprecise.

The New Way (Semi-Supervised Learning):
The authors of this paper developed a new mathematical "detective tool" (a Semi-Supervised Estimator) that does two things at once:

It starts with the Gold Standard answers from the small group to get a solid baseline.
It then cleverly "augments" (boosts) that baseline by using the Clues from the massive group of 113,000 people.

Think of it like this: You have a very accurate but tiny map of a city (the labeled data). You also have a huge, slightly blurry satellite photo of the whole city (the unlabeled data with surrogate clues). The new method figures out how to overlay the blurry photo onto the accurate map to fill in the gaps, making the final map much sharper and more reliable without needing to manually draw every single street.

How It Works (The "Magic" Trick)

The method uses a "working model." Imagine you have a rough guess about how the "Clues" relate to the "Truth."

Step 1: You calculate the risk using only the small, perfect group.
Step 2: You calculate the risk using the huge group with the imperfect clues.
Step 3: The method looks at the difference between these two calculations. It uses the huge group to "correct" the small group's estimate. Even if your guess about how the clues relate to the truth isn't perfect, the math shows that this correction still makes the final result much better than using the small group alone.

The authors even created a "Super-Tool" that combines two different ways of guessing the relationship between clues and truth, making the result even stronger.

What They Found

The researchers tested this tool in two ways:

The Simulation (The Practice Run): They created a fake world of computer-generated patients where they knew the real answer. They found that their new method was significantly more precise than the old method. It reduced the "wiggle room" (uncertainty) in the results by about 40% to 50%. Even when the "clues" were imperfect, the method still worked well.
The Real World Test (Type 2 Diabetes): They applied this to real data from a US health system involving over 115,000 patients.
- They wanted to see how factors like age, gender, and race affect the risk of getting diabetes.
- The old method (using only the 1,600 manually reviewed patients) gave them an answer, but with a wide margin of error.
- The new method (using all 115,000 patients) gave them an answer that was much sharper. For example, the precision of their estimate for the effect of age improved by nearly 80% compared to the old method.

The Bottom Line

This paper doesn't claim to cure diabetes or change how doctors treat patients today. Instead, it offers a better way to do the math when studying disease risks using electronic records.

It proves that you don't have to throw away the massive amount of "imperfect" data just because you only have a few "perfect" answers. By using this new semi-supervised method, researchers can get much more accurate results without having to spend years manually reading paper charts. It turns a "blurry" picture into a clear one, using the power of the whole dataset rather than just a tiny slice of it.

Technical Summary: Semi-supervised Method for Risk Prediction with Doubly Censored EHR Data

Problem Statement
The paper addresses the challenge of estimating risk effects (regression coefficients) in Electronic Health Record (EHR) data where event times are subject to double censoring (both left and right). In many clinical settings, the true onset time of a disease may precede the first recorded visit (left censoring) or remain unobserved at the end of follow-up (right censoring). Furthermore, obtaining "gold-standard" event times often requires labor-intensive manual chart reviews, resulting in a scenario where only a small subset of patients (the labeled set) has accurate outcome data, while the vast majority (the unlabeled set) does not.

While surrogate outcomes (e.g., time to first diagnostic code) are available for the entire cohort, they are prone to misclassification. Relying solely on the small labeled set yields high variability, while using error-prone surrogates directly introduces bias. Existing semi-supervised learning (SSL) methods are effective for binary or right-censored outcomes but do not accommodate the complexities of double censoring.

Methodology
The authors propose a novel semi-supervised learning (SSL) framework for risk prediction under double censoring, utilizing a semiparametric linear transformation model:
$h(T) = -\beta'Z + \varepsilon$
where $T$ is the event time, $Z$ is a vector of covariates, $h$ is an unknown increasing function, and $\varepsilon$ follows a known distribution.

The estimation procedure involves two main steps:

Supervised Estimation: A supervised estimator ( $\hat{\beta}_{SL}$ ) is derived using only the labeled data ( $D_l$ ) via a nonparametric maximum likelihood approach employing an Expectation-Maximization (EM) algorithm with subject-specific independent Poisson variables.
Augmentation: The supervised estimator is augmented using information from the large unlabeled set ( $D_u$ $D_{u}$ ) containing surrogate outcomes ( $T^*$ $T^{*}$ ). The authors assume a working model for the surrogate $T^*$ $T^{*}$ with parameters $\theta = (\gamma, \alpha)$ $θ = (γ, α)$ . Two specific working models are proposed:
- Model 1 (Same Structure): Assumes $T^*$ follows the same semiparametric linear transformation model as $T$ (Eq. 4).
- Model 2 (Flexible Structure): Assumes a more flexible transformation model where the error distribution is unknown and potentially correlated with the true error (Eq. 5). This model utilizes a convex loss function combining a logistic model for left-censoring probability and a Cox model for the right-censored portion.

The final semi-supervised estimator ( $\hat{\beta}_{SSL}$ ) is constructed by adjusting $\hat{\beta}_{SL}$ with a term proportional to the difference between the estimator of the surrogate parameter $\gamma$ derived from the combined data ( $\bar{\gamma}$ ) and the labeled data ( $\hat{\gamma}$ ). A combined estimator ( $\hat{\beta}_{SSL3}$ ) is also proposed by stacking the augmentation terms from both working models to maximize efficiency.

Key Contributions

Methodological Novelty: The paper fills a gap in the literature by developing SSL methods specifically for doubly censored data, a scenario previously unaddressed by existing SSL frameworks.
Theoretical Validity: The authors establish the asymptotic properties of the proposed estimators, proving consistency and asymptotic normality. They demonstrate that the SSL estimators are more efficient than the supervised estimator ( $\hat{\beta}_{SL}$ ) regardless of whether the working models for the surrogate are correctly specified.
Efficiency Gain: The combined estimator is shown to be at least as efficient as estimators based on single working models.
Robustness: The framework is designed to handle the high missingness rate of true labels (where the missing rate approaches 1), rendering standard missing-data methods inapplicable.

Results

Simulation Studies: Extensive simulations with varying sample sizes and error distributions (including cases where the working model is misspecified) show that the proposed SSL estimators significantly reduce the standard error compared to supervised methods. The relative efficiency (RE) of the SSL estimators ranged from approximately 1.35 to 1.52 compared to the supervised estimator. The combined estimator ( $\hat{\beta}_{SSL3}$ ) consistently achieved the highest efficiency.
Real Data Application: The method was applied to EHR data from a US health system to study risk factors for Type 2 Diabetes (T2D). The dataset included 115,236 patients, with only 1,613 having manually reviewed (labeled) onset times. The SSL estimators yielded substantially smaller standard errors than the supervised estimator, with relative efficiencies reaching up to 1.81. The results confirmed that higher enrollment age is associated with later T2D onset, while male gender and non-white race are associated with higher risk.

Significance and Claims
The authors claim that their method provides a principled way to integrate limited gold-standard labels with large-scale surrogate data under double censoring. The primary significance lies in the substantial improvement in estimation efficiency, which reduces the reliance on costly and labor-intensive manual chart reviews. By leveraging the abundant unlabeled EHR data, the method makes the study of disease risk factors in real-world settings more feasible and statistically robust. The paper modestly notes that while the efficiency gain depends on the strength of the association between the surrogate and the true outcome, the proposed framework remains consistent even under model misspecification. Future work is suggested to extend the framework to accommodate covariate-dependent censoring and missing-at-random (MAR) mechanisms.

The Old Way vs. The New Way

How It Works (The "Magic" Trick)

What They Found

The Bottom Line

Technical Summary: Semi-supervised Method for Risk Prediction with Doubly Censored EHR Data

More like this