Statistical uncertainty explains the poor agreement in polygenic scoring for type 2 diabetes

This paper demonstrates that the poor agreement among polygenic scores for type 2 diabetes is fully explained by statistical uncertainty, and proposes that incorporating individual-level uncertainty estimates improves risk prediction accuracy and the identification of high-risk individuals.

The Gist

The Big Problem: Too Many Different Answers

Imagine you go to a doctor to check your risk of getting Type 2 Diabetes. You hand them a DNA test. But instead of one clear answer, the doctor pulls out five different calculators (Polygenic Scores, or PGS) to predict your risk.

• Calculator A says: "You are in the top 2% of risk! You are very likely to get diabetes."
• Calculator B says: "You are in the bottom 50%. You are very safe."

This is the current problem in genetic medicine. Different "calculators" often give completely different answers for the same person. This makes doctors and patients confused. If the tools disagree so much, how can we trust them to make life-changing decisions?

The Discovery: It's Not a Broken Calculator; It's "Static"

The researchers in this paper asked: Why do these calculators disagree?

They discovered that the disagreement isn't because the calculators are broken or because the DNA is wrong. It's because of statistical uncertainty, or what we might call "genetic static."

Think of a polygenic score like trying to listen to a radio station in a car while driving through a tunnel.

• The Signal: Your true genetic risk.
• The Static: The uncertainty in how the scientists calculated the score.

Because the math used to build these scores isn't perfect, there is a "fuzzy zone" around every person's score.

• Person A has a score of 90, but the static is low. The "fuzzy zone" is tight. We are very confident they are high risk.
• Person B also has a score of 90, but the static is high. The "fuzzy zone" is huge. They might actually be a 70 or a 110. We aren't sure.

The paper found that this "static" explains why the calculators disagree. When the static is high, Calculator A might say "High Risk" while Calculator B says "Low Risk" just because they are listening to the static differently.

The Solution: Don't Just Look at the Number; Check the "Confidence"

The researchers proposed a new way to use these scores. Instead of just looking at the number (the point estimate), we should look at the confidence interval (the size of the fuzzy zone).

They created a system to categorize people into three groups:

1. High Confidence: The static is low. The calculators all agree. (e.g., "We are 99% sure this person is high risk.")
2. Medium Confidence: There is some fuzziness. The calculators mostly agree, but not perfectly.
3. Low Confidence: The static is huge. The calculators are shouting different things. We can't trust the result yet.

The Results: Trusting the "High Confidence" Group

When the researchers looked at the people in the High Confidence group, they found something amazing:

• Agreement: Almost all the different calculators agreed on these people.
• Reality: These people were actually much more likely to develop Type 2 Diabetes in real life compared to people who just had a "high number" but low confidence.

The Analogy:
Imagine a weather forecast.

• Scenario 1: A meteorologist says, "It will rain tomorrow," but their data is shaky and they are only 50% sure. You might ignore it.
• Scenario 2: A meteorologist says, "It will rain tomorrow," and their data is rock solid with 99% certainty. You will definitely bring an umbrella.

This paper says: Stop treating all "High Risk" scores the same. Only trust the ones where the "meteorologist" is 100% sure.

The Catch: The "Static" is Louder for Some People

The researchers also found a sad truth about fairness.

• Most of the big genetic studies were done on people of European ancestry. The "radio signal" is clear for them, so the static is low, and the confidence is high.
• For people of African or other ancestries, the "radio signal" is weaker because the studies didn't include enough people like them. The static is much louder.

This means that for people of non-European ancestry, it is much harder to get a "High Confidence" score. If we only treat the "High Confidence" people, we might accidentally leave out many high-risk people from diverse backgrounds, making health inequality worse.

The Bottom Line

1. The Problem: Different genetic risk scores often disagree because of mathematical "noise" (uncertainty).
2. The Fix: We should measure how "noisy" the score is. If the noise is low, we can trust the score. If the noise is high, we should be careful.
3. The Benefit: People with "low noise" (high confidence) scores are the ones most likely to actually get the disease, making them the best candidates for early prevention.
4. The Warning: We need to fix the "noise" for diverse populations so that everyone gets a fair, clear reading, not just those who look like the people in the original studies.

In short: It's not just about what the score says, but how sure we are that the score is right.

Technical Summary

1. Problem Statement

Polygenic scores (PGS) are increasingly used to predict genetic risk for complex diseases like Type 2 Diabetes (T2D). However, a major barrier to clinical implementation is the poor individual-level agreement between different PGS models.

• The Issue: Different PGS models, even those with similar population-level accuracy (AUC), often assign drastically different risk classifications to the same individual. For example, an individual might be classified as "high risk" by one model but "low risk" by another.
• The Gap: It was previously unclear whether this disagreement stemmed from fundamental biological differences in how models capture risk or from statistical noise. Without understanding the source of this variability, clinicians cannot reliably determine which model to trust or how to interpret conflicting results.

2. Methodology

The authors utilized a large, diverse cohort from the All of Us Research Program (AoU) (N = 223,015) and replicated findings in the Penn Medicine Biobank (PMBB).

• Data Sources:
  • AoU v7: Included 24,395 T2D cases and 198,080 controls.
  • Population Stratification: Analyses were primarily restricted to individuals genetically similar to European ancestry (EUR-like) to minimize population-specific bias, with secondary analyses across African (AFR-like) and Admixed American (AMR-like) groups.
• PGS Construction:
  • Reference Models: Created novel T2D PGS using PRS-CS (for EUR-like) and PRS-CSx (multi-ancestry) leveraging the largest available GWAS meta-analyses.
  • Published Models: Downloaded and standardized 57 published T2D PGS from the PGScatalog.
  • Filtering: Selected published PGS with population-level accuracy equivalent to the reference model using a Region of Practical Equivalence (ROPE) (±0.02 around the AUC).
• Uncertainty Estimation:
  • Leveraged the Bayesian nature of PRS-CS/PRS-CSx to generate 100 posterior samples of variant weights per individual.
  • Calculated individual-level credible intervals (e.g., 95% credible range) for each person's PGS based on the variance of these posterior samples.
  • Defined "PGS High Risk Confidence" as the probability that an individual's entire posterior distribution lies above a specific risk threshold (e.g., top 2%).
• Validation Metrics:
  • Empirical Range: The spread of risk scores across the filtered published PGS for a single individual.
  • Predicted Range: The credible interval derived from the single reference PGS.
  • Risk Stability: The percentage of published PGS that agree with the reference model's high-risk classification for a given individual.

3. Key Contributions

1. Attribution of Disagreement: The study demonstrates that the poor agreement between different PGS models is fully explained by statistical uncertainty in the estimation of PGS weights, rather than fundamental biological discrepancies.
2. Single-Model Sufficiency: It shows that the uncertainty estimates from a single, high-quality PGS (using Bayesian posterior sampling) are sufficient to approximate the variability observed across multiple published PGS models.
3. Confidence-Based Stratification: The authors propose a new framework for clinical risk stratification that incorporates risk confidence. Individuals with "high confidence" (narrow credible intervals crossing the threshold) exhibit significantly higher agreement across all models compared to those with "medium" or "low" confidence.
4. Generalizability: The findings were validated across other cardiometabolic traits, specifically Body Mass Index (BMI) and Coronary Heart Disease (CHD), though calibration varied by trait.

4. Key Results

• Uncertainty Explains Variability:

  • There was a strong linear correlation (slope ≈ 0.96) between the predicted range (from a single PRS-CS model) and the empirical range (across 5 published models) for T2D.
  • This confirms that the "noise" causing disagreement is statistical uncertainty inherent in the weight estimation process.

• Risk Stability by Confidence Level:

  • Among individuals in the top 2% risk percentile:
    • High Confidence (>95% probability of being high risk): Showed 89% agreement with other published PGS models.
    • Medium-High Confidence (75–95%): Showed 66.4% agreement.
    • Medium Confidence (50–75%): Showed 38.5% agreement.
  • This indicates that filtering for high-confidence individuals effectively selects a subgroup where risk assessment is robust and consistent across models.

• Clinical Relevance (T2D Onset):

  • High-confidence individuals had significantly higher T2D incidence rates (33.1%) compared to medium-high (26.5%) and medium (23.7%) confidence groups.
  • They also exhibited lower age of onset and higher clinical risk factors (family history, glucose levels).
  • Note: While confidence stratification identified higher-risk groups, adding confidence as a covariate to a standard PGS model did not significantly improve prediction beyond the PGS point estimate itself, likely due to high correlation between the two.

• Population Disparities:

  • Ancestry Bias: The method highlighted health inequities. AFR-like individuals had larger uncertainty intervals and lower risk stability compared to EUR-like individuals.
  • Only <20 AFR-like individuals were classified as "high confidence" (vs. 283 EUR-like), suggesting that current GWAS sample sizes for non-European populations result in higher uncertainty, potentially excluding these groups from high-confidence clinical interventions.

• Trait Specificity:

  • BMI: Showed excellent calibration (slope = 1.0) between predicted and empirical ranges.
  • CHD: Showed initial miscalibration (slope = 0.71) due to higher missing heritability and model variance. Calibration improved significantly (slope = 1.1) only after applying stricter ROPE filtering (±0.01) to the published models.

5. Significance and Implications

• Clinical Implementation: The study provides a practical solution to the "which PGS to use?" dilemma. Instead of averaging multiple scores, clinicians can use a single, well-calibrated PGS with its associated uncertainty estimates to identify patients with robust risk profiles.
• Risk Communication: Reporting credible intervals or confidence levels alongside point estimates can help clinicians and patients understand the reliability of a genetic risk prediction.
• Health Equity: The findings underscore a critical limitation: uncertainty is higher in underrepresented populations. This suggests that without larger, diverse GWAS, PGS-based confidence stratification may inadvertently exacerbate health disparities by excluding minority populations from "high-confidence" clinical pathways.
• Future Directions: The authors suggest that future PGS methods should focus on better uncertainty quantification (e.g., moving beyond PRS-CS to methods like QR-PRS or PredInterval) and increasing sample sizes in diverse ancestries to reduce individual-level uncertainty.

In summary, the paper reframes the problem of PGS disagreement not as a failure of the models, but as a statistical feature that can be quantified and leveraged to improve the precision and reliability of genetic risk prediction.