Validating folding energy estimates as a method for variant interpretation

This study validates the use of FoldX-based folding energy estimates for variant interpretation by demonstrating that, despite moderate overall correlation coefficients, systematic analysis of mega-scale data reveals a strong linear relationship that can be refined by aggregating structural estimates and identifying outliers, thereby establishing a robust framework for flagging low-confidence predictions and improving protein stability assessments.

The Gist

The Big Picture: Why Do We Need This?

Imagine your body is a massive library of instruction manuals (your DNA). Sometimes, a typo happens in these manuals—a genetic variant. Most of the time, we know if a typo is harmless or if it breaks the book. But there are thousands of "Variants of Uncertain Significance" (VUS). These are typos where we don't know: Is this a harmless spelling mistake, or does it destroy the instruction manual?

Scientists have been trying to use computer programs to predict if a typo will break the protein (the machine built from the manual). One popular program is called FoldX. It tries to calculate how much "energy" it takes for a protein to fold correctly. If the energy is too high, the protein might misfold and break, causing disease.

The Problem: For years, scientists have argued about FoldX. Sometimes it works great; other times, it's all over the place. It's like a weather forecaster who is right 90% of the time in summer but only 30% of the time in winter. Because the results are so inconsistent, many doctors and researchers don't trust it enough to use it for diagnosing patients.

The Experiment: A "Mega-Scale" Stress Test

The authors of this paper decided to stop arguing and start testing. They used a massive dataset (from a study by Tsuboyama et al.) that contained experimental data on over 1,000 mutations across seven different proteins.

Think of this as taking FoldX and throwing it into a giant obstacle course with thousands of different hurdles, rather than just testing it on a few easy ones.

What They Found: The "Outlier" Problem

When they first looked at the data, the results looked messy. The correlation between what FoldX predicted and what actually happened in the lab was weak (about 0.30). It looked like the computer was guessing.

But then, they found the secret.

They realized that the "messy" results were being dragged down by a tiny handful of bad apples.

• The Analogy: Imagine you are grading a class of 100 students. 95 of them score between 80 and 90. But 5 students scored 0 because they fell asleep in class. If you average the whole class, the grade looks terrible. But if you realize those 5 students were just "outliers" (maybe they were sick or the test was broken for them), the average of the rest of the class is actually quite good.

The paper found that a very small number of specific amino acids (the building blocks of proteins) were causing the computer to crash or give wild, wrong answers. These were usually "outlier" residues.

Once they identified and set aside these problematic "bad apples," the relationship between the computer's prediction and the real-world experiment became clear and linear. The computer wasn't bad; it just needed help ignoring the noise.

The Solution: The "Median" Strategy

The researchers also noticed that for a single protein, there are often many different 3D models (structures) available. Sometimes the protein is frozen in one pose, sometimes another.

• The Analogy: Imagine trying to guess the height of a person by looking at photos taken from different angles. One photo makes them look tall because they are standing on a box; another makes them look short because they are slouching.
• The Fix: Instead of trusting just one photo, the team took the median (the middle value) of all the different photos. By averaging out the weird angles, they got a much more accurate picture of the person's true height.

By taking the "median" prediction across all available protein structures, they boosted the accuracy significantly. In some cases, the computer's predictions were almost as good as the experimental data itself (which is the gold standard).

Why Do the "Bad Apples" Happen?

The team dug deeper to ask: Why do these specific residues confuse the computer?

They found that the problematic spots are usually in tight, rigid parts of the protein.

• The Analogy: Imagine a crowded dance floor. In the open spaces, people can move easily. But in the tight corners, if one person tries to move, they bump into everyone else.
• The Result: When the computer tries to simulate a mutation in these tight corners, it struggles to "repack" the atoms properly. It overestimates how much energy is needed, leading to a wrong prediction.

They even developed a way to spot these "tight corners" in advance using a mathematical model (Elastic Network Model). This means they can now flag a prediction as "Low Confidence" before a doctor even looks at it.

The Takeaway: Trusting the Tool Again

The Conclusion:
This paper is a huge validation for FoldX. It proves that the tool is actually very powerful for predicting how mutations affect protein stability. The reason it looked unreliable before wasn't because the tool was broken, but because:

1. A few specific "tricky" spots were skewing the data.
2. Scientists were looking at single structures instead of averaging many.

Why This Matters for You:

• Better Diagnosis: Doctors can now use these computer predictions to help interpret genetic test results for patients with rare diseases.
• Drug Design: If we know exactly which mutations break a protein, we can design drugs to fix them.
• Efficiency: Instead of running expensive and slow lab experiments for every single mutation, we can use these fast, accurate computer screens to filter out the dangerous ones first.

In a Nutshell:
The authors took a tool that everyone thought was "okay but unreliable," cleaned up the data, ignored the outliers, and showed that it's actually a super-accurate crystal ball for understanding how genetic mutations break our bodies. They just had to teach us how to look at the data correctly.

Technical Summary

1. Problem Statement

The interpretation of Variants of Uncertain Significance (VUS) remains a critical bottleneck in genomic analysis. While statistical models can predict pathogenicity, they often lack mechanistic insight and suffer from training data biases toward specific subpopulations.

• Biophysical Context: Protein misfolding is a primary mechanism for loss of gene activity, responsible for approximately two-thirds of disease-causing missense mutations.
• The Challenge: Computational tools like FoldX are widely used to predict the change in folding free energy ($\Delta\Delta G$) caused by mutations. However, their predictive power has been questioned due to highly variable correlation coefficients reported in literature (ranging from 0.2 to 0.8) and the sensitivity of results to the choice of initial protein structure.
• Goal: The authors aim to systematically validate FoldX predictions at a mega-scale to determine if these tools can reliably support variant interpretation and to understand the sources of prediction error.

2. Methodology

The study utilized a fully automated, high-performance computing pipeline to perform computational saturation screens against a massive experimental dataset.

• Data Source: The study leveraged the Tsuboyama et al. (2023) mega-scale experimental dataset, which contains over 1,000 well-validated substitutions across seven proteins with high experimental agreement ($r > 0.75$).
• Computational Pipeline ("Mutein"):
  • Structure Retrieval: Retrieved all available PDB structures and AlphaFold models for target genes via UniProt and AlphaFold DB.
  • Preprocessing: Ran FoldX RepairPDB (5 iterations) to minimize steric clashes and optimize residue orientations.
  • Mutation Simulation: Executed FoldX PositionScan to calculate $\Delta\Delta G$ for every possible amino acid mutation on every residue of every structure.
  • Alignment & Correction: Developed a multi-step alignment protocol to map PDB residues to the canonical UniProt reference sequence. Crucially, they applied a thermodynamic cycle correction to account for amino acid differences between the experimental construct and the reference sequence, ensuring all energies were relative to the same baseline.
• Statistical Analysis:
  • Outlier Detection: Used linear regression to identify "problematic" substitutions (those >2 standard deviations from the line of best fit).
  • Aggregation: Compared individual structure predictions against the median prediction across all available structures for a given protein.
  • Mechanistic Investigation: Used Elastic Network Modeling (ENM) and Gaussian Network Models to analyze residue flexibility and motion modes to explain why certain residues are poorly predicted.

3. Key Contributions

• Systematic Validation at Scale: Provided one of the largest validations of FoldX against experimental data, moving beyond single-protein case studies to a multi-protein cohort.
• Decoupling Correlation from Predictive Power: Demonstrated that low correlation coefficients (e.g., $r \approx 0.30$) are often artifacts of a small number of extreme outliers rather than a failure of the underlying linear relationship.
• Identification of "Problematic" Residues: Identified that a small subset of residues (often 4–5 per protein) accounts for the majority of prediction errors (up to 79% of outliers).
• Structural Determinants of Error: Linked prediction failures to specific structural features:
  • Outliers are associated with tightly constrained residues (identified via fast modes of motion in Elastic Network Models).
  • Errors often stem from poor repacking/minimization of bulky aromatic residues (Tyrosine, Phenylalanine) and Histidine in rigid environments.
• Protocol Optimization: Established that aggregating predictions via the median across multiple structures significantly improves accuracy, approaching the theoretical limit set by experimental reproducibility.

4. Key Results

• Linear Relationship with Outliers: For proteins like PIN1, FYN, and Spg, the raw correlation between FoldX predictions and experimental data was mediocre ($r \approx 0.30–0.60$). However, visual inspection revealed a clear linear trend obscured by a few extreme outliers.
• Impact of Outlier Removal: When the small number of "problematic" residues (identified as >2 SD from the regression line) were excluded, the correlation improved significantly.
  • Example (PIN1): Correlation increased from 0.30 (all data) to 0.61 (median of non-outliers).
  • Example (Spg): Correlation increased to 0.72.
• Structure Aggregation: Taking the median $\Delta\Delta G$ across multiple structures for a single protein consistently outperformed predictions from any single structure, reducing variance and improving correlation coefficients.
• Confidence Intervals: The study established empirical confidence intervals for predictions based on structural variation:
  • Non-outlier substitutions: $\pm 0.27$ to $+0.35$ kcal/mol (PIN1).
  • Outlier substitutions showed notably broader variance, allowing them to be flagged as low-confidence.
• Mechanistic Insight: Elastic Network Modeling revealed that residues with high variance in prediction errors are located in tightly constrained regions (fast modes of motion). These residues are difficult for FoldX to repack correctly after mutation, leading to overestimates of instability.
• Global Performance: When applied to an expanded set of ~200 proteins (58 with sufficient data), the median-aggregated approach yielded correlations approaching the experimental reproducibility limit ($r \approx 0.75$). However, pooled global correlations were lower ($r=0.45$) due to protein-specific slope differences, suggesting FoldX is better at ranking mutations within a protein than comparing absolute energies across proteins.

5. Significance and Conclusion

• Validation of FoldX: The study validates FoldX as a robust tool for prospective calculation of misfolding energies, provided that data is aggregated across structures and outliers are managed.
• Framework for Variant Interpretation: The authors propose a framework where low-confidence predictions (outliers) are flagged automatically. This allows clinicians and researchers to trust high-confidence predictions while treating low-confidence ones with caution or seeking alternative methods.
• Mechanistic Understanding: By identifying that errors arise from rigid, tightly packed residues, the paper offers specific targets for improving force fields or minimization protocols (e.g., using ensemble-based methods like Rosetta or dynamics simulations like BioEmu for these specific regions).
• Future Directions: The work supports the creation of pre-calculated databases of misfolding energies to complement existing pathogenicity scores. It suggests that future improvements should focus on handling structural ensembles and better repacking of bulky residues in constrained environments.

In summary, the paper resolves the contradiction between studies reporting poor FoldX performance and those showing high accuracy by demonstrating that outlier residues distort global correlations, but the underlying predictive power is strong and reliable when these specific structural challenges are identified and mitigated.