📊 statistics

Accounting for Heavy Censoring in Evaluating the Risk Stratification Abilities of Existing Models for Time to Diagnosis of Huntington Disease

This study externally validates and compares four existing risk stratification models for Huntington disease using the ENROLL-HD dataset and censoring-appropriate metrics, demonstrating that while the Multivariate Risk Score (MRS) model performs best, the simpler Prognostic Index Normed (PIN) model offers comparable utility with fewer variables, and highlighting that failing to account for high censoring rates leads to underpowered clinical trial designs.

Original authors: Kyle F. Grosser, Abigail G. Foes, Stellen Li, Vraj Parikh, Tanya P. Garcia, Sarah C. Lotspeich

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kyle F. Grosser, Abigail G. Foes, Stellen Li, Vraj Parikh, Tanya P. Garcia, Sarah C. Lotspeich

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Predicting the Unpredictable

Imagine Huntington's Disease (HD) as a slow-moving, invisible storm. We know the storm will hit everyone with a specific genetic marker (the "CAG repeat"), but we don't know exactly when the first drop of rain will fall.

For doctors and researchers, knowing the "time of arrival" is crucial. If they want to test a new medicine to stop the storm, they need to find people who are about to get wet soon. If they test the medicine on people who won't get wet for another 20 years, the trial will take forever, cost a fortune, and might fail to show if the medicine works.

This paper is about finding the best "Weather Forecast" model to predict when that storm will hit, so researchers can pick the right patients for their clinical trials.

The Problem: Too Many Forecasts, Too Much Fog

There are currently four different "Weather Models" (mathematical formulas) that scientists use to predict when a patient will be diagnosed with HD:

Langbehn: The old-school model.
CAP: A slightly newer model.
MRS: A complex model that looks at many details.
PIN: A balanced model.

The Issue:

The Fog (Censoring): In medical studies, most patients don't get diagnosed while the study is running. They drop out, or the study ends before they get sick. In statistics, this is called "censoring." It's like trying to predict a storm, but 80% of your weather stations stop sending data before the rain starts.
The Mistake: Previous studies tried to compare these models, but they made two big mistakes:
1. Cheating: They tested the models on the exact same data used to build them (like a student taking a test on the answer key they just wrote).
2. Ignoring the Fog: They used standard scoring methods that assume they knew the diagnosis date for everyone, even though they didn't. This made the scores look better than they really were.

The Solution: A New Test Drive

The authors decided to put these four models through a rigorous test drive using a massive, brand-new dataset called ENROLL-HD (which was not used to build any of the models).

They used two special tools to score the models that could handle the "fog" (the missing data):

Uno's C Statistic: Think of this as a ranking game. If you have two patients, Patient A and Patient B, and Patient A gets sick sooner than Patient B, did the model correctly guess that Patient A was "riskier"?
Censored ROC Curves: Think of this as a target practice. Can the model separate the people who will get sick in the next 3 years from those who won't?

The Results: Who Won the Race?

After running the numbers, here is how the models stacked up:

🥇 The Gold Medalist (MRS): The Multivariate Risk Score was the most accurate. It looked at the most data points (age, genetics, motor skills, brain scans, cognitive tests, etc.). It was the best at sorting patients into "high risk" and "low risk."
🥈 The Silver Medalist (PIN): The Prognostic Index Normed came in a very close second. It used fewer data points (just 4 variables instead of 8) but performed almost as well as the Gold Medalist.
🥉 The Bronze (CAP & Langbehn): These models only looked at age and genetics. They were okay, but they missed out on important details like how well a patient could walk or think.

The "Simple vs. Complex" Trade-off:
The authors found a sweet spot. The MRS model is the most accurate, but it requires patients to take many tests (like a full medical workup). The PIN model is slightly less accurate but much simpler, requiring only a few tests.

Analogy: If you want the most precise weather forecast, you need a supercomputer with satellite data (MRS). If you just need a good enough forecast to decide whether to bring an umbrella, a simple barometer (PIN) works great and is much easier to carry.

The Real-World Impact: Saving Time and Money

The most important part of the paper is how this helps design clinical trials.

Imagine you want to test a new drug. You need to find 100 people who will get diagnosed with HD within 3 years.

The Old Way: If you pick patients randomly, you might have to enroll 500 people to find those 100 "high-risk" ones. That's expensive and slow.
The New Way (Sample Enrichment): Using the MRS or PIN models, you can calculate a "Risk Score" for everyone. You then say, "We will only enroll people with a Risk Score above 8.96."
- Suddenly, you might only need to enroll 150 people to find your 100 high-risk candidates.
- The Catch: The authors showed that if you use the old methods (ignoring the "fog" of missing data), you might think you only need 70 people. If you run a trial with 70 people, you will likely fail because you didn't have enough "events" (diagnoses) to prove the drug worked.

The Takeaway

Don't trust the old scores: Previous comparisons of these models were flawed because they ignored missing data.
MRS is the champion: If you have all the data, use the MRS model. It's the most accurate.
PIN is the smart alternative: If you don't have time for all the tests, use the PIN model. It's nearly as good and much easier to use.
Better models = Better trials: By using these models correctly to pick the right patients, we can run smaller, cheaper, and faster clinical trials to find cures for Huntington's Disease sooner.

In short: The authors fixed the ruler we use to measure risk, found the best tool for the job, and showed us how to use it to build better clinical trials without wasting time or money.

1. Problem Statement

Huntington Disease (HD) is a fully penetrant neurodegenerative disorder where clinical diagnosis relies on motor impairment assessments. Designing preventative clinical trials (aimed at delaying diagnosis in pre-symptomatic, genetically confirmed patients) requires accurate risk stratification to enrich study populations with patients closer to diagnosis. This reduces the required sample size and cost.

Currently, four major time-to-diagnosis models exist:

Langbehn (Parametric, logistic distribution).
CAG-Age Product (CAP) (Parametric, accelerated failure time).
Multivariate Risk Score (MRS) (Semiparametric, Cox model with many covariates).
Prognostic Index Normed (PIN) (Semiparametric, Cox model with fewer covariates).

Critical Gaps Identified:

Lack of Systematic Comparison: Previous studies compared subsets of these models or failed to compare all four simultaneously.
Double-Dipping: Most evaluations used the same dataset for model development and validation (e.g., PREDICT-HD), leading to over-optimistic performance estimates.
Ignoring Heavy Censoring: HD observational studies often have >75% right-censoring (patients remain undiagnosed by study end). Traditional metrics like standard ROC curves and Harrell's C statistic assume complete data or discard censored cases, leading to biased performance assessments and potentially underpowered clinical trials.

2. Methodology

Data Source

The authors utilized data from the ENROLL-HD study, a massive, global observational cohort ( $N=3113$ after filtering for CAG $\ge$ 40 and pre-diagnosis status). Crucially, ENROLL-HD was independent of the datasets used to develop the four models, allowing for true external validation. The dataset exhibited a 77% right-censoring rate.

Models Evaluated

The study compared the four models under two scenarios:

External Validation: Applying original published parameters (from training datasets like PREDICT-HD) to ENROLL-HD.
Model Updating: Re-estimating model coefficients using ENROLL-HD data while retaining the original model structures (covariates and distributional assumptions).

Performance Metrics (Censoring-Adjusted)

To address the high censoring rate, the authors employed two specific metrics:

Uno's Concordance (Uno's C) Statistic: A rank-based measure that uses Inverse Probability of Censoring Weights (IPCW) to account for censored observations. It estimates the probability that a model correctly ranks a pair of patients (diagnosed earlier vs. later) based on their log-risk scores.
Kaplan-Meier Adjusted ROC Curves: Time-dependent ROC curves that adjust for censoring to calculate the Area Under the Curve (AUC). This assesses the model's ability to separate patients diagnosed by time $t$ from those not yet diagnosed.

Application to Trial Design

The authors demonstrated how to use these models for sample enrichment:

Deriving an optimal log-risk score threshold ( $q^*$ ) by maximizing Youden's J statistic on the censoring-adjusted ROC curve.
Calculating sample size requirements for preventative trials based on the expected diagnosis rate of patients exceeding this threshold.

3. Key Results

Model Performance Comparison

Ranking: Across both external validation and model updating, the performance ranking was consistent:
1. MRS: Best performance (Global Uno's C $\approx$ 0.84–0.88).
2. PIN: Second best (Global Uno's C $\approx$ 0.82–0.84), only ~3% worse than MRS.
3. CAP & Langbehn: Tied for third (Global Uno's C $\approx$ 0.80), performing ~4% worse than PIN.
Impact of Updating: Re-estimating parameters on ENROLL-HD data did not significantly improve performance (differences were <1–6%), suggesting the original models are robust and transportable.
Time Dependency: All models showed a gradual decline in predictive accuracy over time (e.g., at 5 years vs. 1 year), likely because they rely on baseline covariates only.

Covariate Trade-offs

MRS requires 8 covariates (Age, CAG, DCL, TMS, SDMT, Stroop Word/Color/Interference).
PIN requires 4 covariates (Age, CAG, TMS, SDMT).
CAP/Langbehn require only 2 covariates (Age, CAG).
Finding: The PIN model offers a "sweet spot," providing nearly the same predictive power as the complex MRS model but with half the data requirements.

Clinical Trial Sample Size Implications

Using the MRS model with an optimal threshold for a 3-year trial, the expected diagnosis rate was 42%.
To detect a 50% treatment reduction with 80% power, 77 patients per arm were required.
Critical Finding: Previous studies (e.g., Paulsen et al., 2019) that ignored censoring in their metrics estimated smaller sample sizes (e.g., 67 patients). The authors demonstrate that using non-censoring-adjusted metrics leads to underpowered trials, increasing the risk of failing to detect effective treatments.

4. Key Contributions

First Comprehensive Comparison: Provides the first systematic, head-to-head comparison of all four major HD time-to-diagnosis models using an independent, large-scale dataset.
Methodological Rigor: Highlights and corrects the "double-dipping" and "ignoring censoring" flaws in previous literature by utilizing external validation and IPCW-adjusted metrics (Uno's C and KM-ROC).
Practical Guidance: Offers a clear decision framework for researchers:
- Use MRS if all clinical/cognitive data is available.
- Use PIN if only TMS and SDMT are available (best balance of simplicity and accuracy).
- Use CAP if only Age and CAG are available.
- Avoid Langbehn due to computational complexity and lack of performance advantage over CAP.
Trial Design Impact: Demonstrates that failing to account for heavy censoring in sample size calculations leads to underpowered studies, providing a corrected framework for designing future preventative HD trials.

5. Significance

This paper fundamentally shifts the standard for evaluating HD risk models. By proving that censoring-adjusted metrics are essential for accurate performance assessment, it prevents the deployment of underpowered clinical trials. The identification of the PIN model as a highly efficient alternative to the complex MRS model facilitates the practical implementation of sample enrichment strategies, making preventative HD trials more feasible and cost-effective. The work underscores the necessity of using appropriate statistical methods for survival data with high censoring rates in neurodegenerative disease research.