Benchmarking Heritability Estimation Strategies Across 86 Configurations and Their Downstream Effect on Polygenic Risk Score Performance

This study systematically benchmarks 86 SNP heritability estimation configurations across six tool families and ten method groups, revealing that while heritability estimates vary substantially based on algorithmic choices and standardization, this upstream variability has minimal impact on downstream polygenic risk score performance, suggesting heritability should be treated as a configuration-sensitive parameter rather than a stable scalar.

The Gist

The Big Picture: The "Recipe" Problem

Imagine you are trying to bake the perfect cake (which, in this study, is a Polygenic Risk Score, or PRS). This cake predicts how likely a person is to get a disease like heart disease or diabetes based on their DNA.

To bake this cake, you need a specific ingredient: Heritability. Think of heritability as a "flavor intensity" number. It tells the baker (the computer algorithm) how much of the cake's taste comes from the genetic recipe versus random noise.

The Problem:
In the past, scientists thought there was only one correct number for this flavor intensity. But this paper asks: "What if different bakers use different ovens, different measuring cups, and different scales to measure that same ingredient? Do they all get the same number?"

The answer is a resounding no.

The Experiment: The Great Kitchen Contest

The researchers set up a massive kitchen contest.

• The Ingredients: They used DNA data from 10 different health conditions (like asthma, high cholesterol, and depression) from the UK Biobank (a giant database of 500,000 people).
• The Bakers: They tested 86 different ways (configurations) to measure that "flavor intensity" (heritability). These 86 ways came from 6 different software families (like GCTA, GEMMA, LDAK, etc.).
• The Goal: They wanted to see two things:
  1. How much do these 86 different methods disagree on the flavor number?
  2. Does it actually matter if the number is slightly different when they finally bake the cake (the PRS)?

The Results: Chaos in the Kitchen, Calm in the Cake

1. The Numbers Were All Over the Place

The researchers found that the "flavor intensity" number varied wildly depending on which method you used.

• The Range: The numbers ranged from -0.86 (negative flavor?!) to 2.73 (super intense flavor!). The average was a tiny 0.13.
• The Negative Numbers: Some methods gave negative numbers. The authors explain this isn't a "broken" machine; it's like a scale that is so sensitive to the wind (noise) that it reads negative when there's nothing on it. It just means the signal was too weak for that specific method.
• The Cause: The biggest reasons for the disagreement were the algorithm (the math used) and how they handled the Genetic Relatedness Matrix (a fancy way of saying "how we compare people's DNA to each other"). It's like using a digital scale vs. a spring scale; they both weigh things, but they give different numbers.

The Lesson: Heritability isn't a fixed, universal constant like the speed of light. It's more like a temperature reading that changes depending on which thermometer you use and how you hold it.

2. The Cake Still Tasted the Same

Here is the most surprising part. Even though the "flavor intensity" numbers were wildly different (some negative, some huge), the final cake (the PRS) tasted almost exactly the same.

• The Analogy: Imagine you are driving a car. You have a speedometer that is broken and sometimes says you are going -10 mph, and sometimes says you are going 100 mph. But as long as you are actually driving at a steady 60 mph, you still get to your destination on time.
• The Finding: The researchers found almost zero connection between how high or low the heritability number was and how well the final risk score predicted the disease. Whether they used a "high" heritability number or a "low" one, the prediction accuracy didn't change much.

Why This Matters: The "Report Card" Lesson

Before this study, scientists might have been worried: "Oh no, I used Method A and got a heritability of 0.1, but my colleague used Method B and got 0.5. Who is right? Is my prediction wrong?"

This paper says: Stop worrying about the exact number.

1. Context is King: You can't just report a heritability number like "0.15" and expect it to mean anything. You have to say, "0.15, measured using Method X, with these specific settings." It's like saying "The temperature is 70 degrees," but you must add "using a thermometer in the shade," not "in the sun."
2. Robustness: The good news is that the tools we use to predict disease (PRS) are very tough. They can handle a lot of "noise" in the input numbers. Even if you pick a slightly imperfect way to measure heritability, your final prediction is likely still reliable.
3. Negative Numbers aren't Failures: If a method gives a negative heritability, don't throw it away immediately. It just means that specific method is sensitive to noise. It's a valid data point, not a broken tool.

The Takeaway

Think of heritability estimation like navigating with a map.

• Different map apps (methods) might give you slightly different distances to the destination. One might say 5 miles, another 6 miles.
• However, as long as you are driving toward the destination, it doesn't matter if your map says 5 or 6 miles; you will still arrive at the same place.

The authors' final advice: When scientists report these numbers, they must be transparent about how they got them. But for the people using these numbers to build disease predictions, the system is surprisingly stable. You don't need to find the "perfect" heritability number to get a good prediction; you just need a reasonable one and a clear recipe.

Technical Summary

1. Problem Statement

SNP heritability ($h^2$) is a fundamental parameter in statistical genetics, routinely used as an input for constructing Polygenic Risk Scores (PRS). However, $h^2$ estimates are not fixed quantities; they vary significantly depending on the estimation strategy, software tool, input data type (genotypes vs. summary statistics), preprocessing steps (clumping, pruning), and model assumptions (e.g., constrained vs. unconstrained optimization).

The Gap: While individual heritability estimators have been studied in isolation, there is a lack of systematic benchmarking regarding:

1. How much variation exists across a broad spectrum of estimation configurations.
2. How this upstream variability propagates into downstream PRS performance.
3. Whether specific heritability values predict better PRS accuracy.

2. Methodology

Study Design:
The authors conducted a large-scale benchmark using data from the UK Biobank (European ancestry participants).

• Phenotypes: 10 traits (9 binary, 1 continuous: Body Mass Index).
• Configurations: 86 distinct heritability estimation configurations derived from 6 tool families (GEMMA, GCTA, LDAK, DPR, LDSC, SumHer) organized into 10 method groups.
• Total Estimates: 844 configuration-level estimates (86 configurations $\times$ 10 phenotypes).

Workflow:

1. Data Preprocessing: Genotype data underwent strict QC (MAF > 0.01, HWE, missingness filters). GWAS summary statistics were harmonized.
2. Cross-Validation: Data was split into 5 folds. Heritability was estimated only on the training fold (80%) and the resulting $h^2$ was used to parameterize the PRS model for that specific fold.
3. PRS Construction: Two frameworks were tested:
   • GCTA-SBLUP: Uses $h^2$ to calculate the shrinkage parameter $\lambda$.
   • LDpred2-lassosum2: Uses $h^2$ to parameterize the regularization grid.
4. Evaluation: PRS performance was evaluated on the held-out test fold (20%) using:
   • Null Model: Covariates + PCs only.
   • PRS-Only Model: Raw PRS score.
   • Full Model: PRS + Covariates.
   • Metrics: AUC (binary traits) and $R^2$ (continuous traits).
5. Statistical Analysis:
   • Hyperparameter Contrasts: 11 binary contrasts (e.g., REML vs. HE regression, Clumping vs. No Clumping) were tested using Mann–Whitney U tests to identify drivers of variability.
   • Correlation Analysis: Pearson/Spearman correlations between $h^2$ magnitude and downstream PRS performance.
   • Matched-Input Control: A subset analysis where methods were compared using the maximum number of variants to isolate estimator differences from preprocessing differences.

3. Key Contributions

1. Comprehensive Benchmarking: The study provides the first systematic comparison of 86 heritability configurations across 10 diverse phenotypes, linking upstream estimation directly to downstream PRS utility.
2. Quantification of Variability: It demonstrates that $h^2$ is highly sensitive to configuration, ranging from -0.862 to 2.735 across the benchmark, with 15.8% of estimates being negative.
3. Decoupling of $h^2$ and PRS Performance: A critical finding is that the magnitude of the heritability estimate is not a reliable predictor of downstream PRS accuracy.
4. Interpretation of Negative Estimates: The paper clarifies that negative $h^2$ values are often mathematical artifacts of unconstrained estimators (like Haseman-Elston regression) under low signal-to-noise conditions, rather than evidence of estimator failure.

4. Key Results

A. Variability of Heritability Estimates:

• Range: $h^2$ varied widely (Mean = 0.134, SD = 0.284).
• Drivers of Variation: Ten of eleven analytical contrasts significantly affected $h^2$. The largest effects were driven by:
  • Algorithm Choice: REML-based configurations yielded significantly higher $h^2$ (Mean 0.252) than Haseman-Elston (HE) regression (Mean 0.043).
  • GRM Standardization: Standardized GRMs produced higher estimates than centered GRMs.
  • Clumping/Pruning: These steps consistently reduced $h^2$ estimates.
• Negative Estimates: 133 estimates (15.8%) were negative, concentrated in unconstrained methods (DPR+GEMMA, GEMMA, LDSC). Constrained methods (GCTA) produced zero negative estimates.

B. Downstream PRS Performance:

• Weak Coupling: There was no significant correlation between the magnitude of $h^2$ and PRS test performance across the full benchmark.
  • GCTA-SBLUP: $r = -0.023$ ($p=0.188$)
  • LDpred2-lassosum2: $r = +0.014$ ($p=0.427$)
• Robustness: Despite massive upstream variability in $h^2$ (e.g., $h^2$ ranging from 0.0 to 1.0 for a single phenotype), the resulting test AUC varied by less than 0.07 units for most binary traits.
• Method Comparison: No single heritability estimation family consistently outperformed others in downstream PRS accuracy.

C. Configuration Selection:

• A "delta-constrained" heuristic (selecting configurations with small train-test gaps) identified more stable configurations than "best-train" selection, suggesting that training-only optimization favors overfitting in this context.

5. Significance and Implications

• Reframing Heritability: SNP heritability should be interpreted as a configuration-sensitive modeling parameter rather than a universal, stable scalar. A reported $h^2$ value is meaningless without its full estimation specification (software, algorithm, preprocessing, covariates).
• Reporting Standards: Researchers must report the full estimation context alongside $h^2$ values.
• PRS Robustness: The study offers reassurance to clinical and translational researchers: moderate variations in the heritability input (due to different estimation strategies) do not systematically destabilize downstream PRS performance. This suggests PRS pipelines are robust even when the "optimal" heritability estimation strategy is uncertain.
• Negative Values: Negative heritability estimates should not automatically disqualify a method; they are informative outputs of unconstrained estimators in low-signal regimes and should be evaluated via confidence intervals rather than point estimates alone.

Conclusion:
The paper concludes that while heritability estimation is highly sensitive to analytical choices, the downstream impact on Polygenic Risk Score performance is comparatively robust. This decoupling supports the practical utility of PRS in real-world settings where estimation strategies may vary, provided that configuration choices are transparently reported.