Evaluating FoldX5.1 for MAVISp Stability Data Collection

This study validates the adoption of FoldX5.1 for the MAVISp database by demonstrating high agreement with the previous FoldX5 version across nearly 540,000 variants, supporting a gradual transition strategy that includes new metadata annotations to ensure data compatibility.

The Gist

Imagine you are running a massive library of "Protein Instruction Manuals." These manuals tell scientists how tiny building blocks in our bodies (proteins) work and what happens when a single letter in their code gets changed (a mutation). One of the most important sections in these manuals is the "Stability Check," which predicts if a mutation will make the protein fall apart or stay strong.

For years, this library (called MAVISp) has used a specific calculator tool named FoldX5 to do these stability checks. But recently, the makers of FoldX released an upgraded version, FoldX5.1, which is like a newer, smarter calculator with better math for handling tricky situations (like how proteins stick together or handle water).

The big question for the library curators was: "Should we switch to the new calculator immediately, or will it mess up all our old records?"

Here is what the researchers did to find out, explained simply:

1. The Great "Double-Check" Experiment

To answer the question, the team didn't just guess. They took 539,809 existing protein mutation records (that's a lot of data!) and ran them through both the old calculator (FoldX5) and the new one (FoldX5.1).

Think of it like having two different weather forecasters predict the temperature for the same 119 cities.

• The Result: For almost every city (protein), the two forecasters agreed perfectly. If the old one said "It will be hot," the new one said "It will be hot."
• The Score: They agreed 93% of the time on the exact numbers and 81% of the time on the general category (Stable, Unstable, or Neutral).

2. The "Troublemaker" Proteins

However, just like in any big group, there were a few outliers. Three proteins—NUPR1, TSC1, and TMEM127—were the "rebellious teenagers" of the dataset. The two calculators gave them very different answers.

The researchers investigated why:

• The "Blurry Photo" Theory: For two of these proteins, the disagreement happened mostly in areas where the 3D model of the protein was "blurry" or low-quality (like trying to predict the weather in a foggy valley). When the map is unclear, even the best calculators struggle to agree.
• The "New Math" Theory: For the third protein, the disagreement happened with specific types of mutations (involving aromatic amino acids). The new calculator (FoldX5.1) has updated rules for how these specific parts stick together, so it gave a different, likely more accurate, answer than the old one.

3. No "System Glitch"

The team was worried that the new calculator might have a "bias"—like a scale that suddenly weighs everything 5 pounds heavier. They checked this and found no bias. The new calculator didn't just shift all the numbers up or down; it just made specific, smarter adjustments where needed.

4. The Final Verdict: A "Soft Launch"

So, what should the library do?

Instead of shutting down the library to re-calculate every single entry from scratch (which would take years and delay progress), they decided on a Phased Transition:

• Old Entries: The existing records will keep their old "FoldX5" labels for now. They stay in the library as they are.
• New Entries: Any new data or updates will use the "FoldX5.1" calculator immediately.
• Transparency: Every entry will now have a little tag (metadata) saying, "This was calculated with Version 5" or "This was calculated with Version 5.1."

The Takeaway

Think of this like upgrading the software on your phone. You don't need to delete all your old photos and re-take them to get the new camera features. You just start using the new camera for your next photos, while your old photos remain safe and viewable.

The researchers concluded that FoldX5.1 is safe to use. It agrees with the old version almost everywhere, and where it differs, it's usually because it's fixing a mistake or handling a blurry picture better. By switching gradually, the MAVISp library gets smarter without losing its history.

Technical Summary

1. Problem Statement

The MAVISp (Multi-layered Assessment of VarIants by Structure for proteins) framework is a structure-based database used to interpret the mechanistic effects of missense variants, with protein stability prediction being a core analytical layer. As computational tools evolve, database curators face a critical challenge: balancing methodological improvement with database continuity.

Specifically, the MAVISp team needed to decide whether to replace the legacy FoldX5 tool with the updated FoldX5.1 version. The central questions were:

• Does the new version compromise compatibility with existing database entries?
• Is it necessary to recompute all historical data (a resource-intensive process) before adopting the new tool, or can a phased transition occur?
• Do the updates in FoldX5.1 (e.g., improved force fields for $\pi$-stacking, pH dependence, and solvation) introduce systematic biases that would invalidate previous stability classifications?

2. Methodology

The authors conducted a large-scale comparative analysis to evaluate the compatibility between FoldX5 and FoldX5.1 within the MAVISp workflow.

• Dataset: The study focused on 119 representative proteins (selected to cover diverse folds) already present in the MAVISp database. This yielded 539,809 shared variants for direct comparison.
• Workflow: The analysis utilized the MAVISp "simple mode" (high-throughput data collection) to reflect standard operational conditions.
  • Structures were processed using the RepairModel step followed by BuildModel-based in silico saturation mutagenesis via MutateX.
  • Predictions were generated using both FoldX5 (legacy) and FoldX5.1 (new).
• Metrics for Agreement:
  • Quantitative: Pearson correlation coefficients ($r$) and Mean Absolute Error (MAE) for predicted folding free energy changes ($\Delta\Delta G$).
  • Qualitative: Cohen's $\kappa$ coefficient and classification accuracy. Predictions were collapsed into three classes based on a $\pm 3.0$ kcal/mol threshold: Stabilizing ($\Delta\Delta G < -3.0$), Neutral ($-3.0 \le \Delta\Delta G \le 3.0$), and Destabilizing ($\Delta\Delta G > 3.0$).
• Outlier Analysis: Proteins showing poor agreement were flagged if they met any of the following: $r < 0.7$, $\kappa < 0.5$, or accuracy $< 75\%$.
• Root Cause Investigation: For outliers, the authors analyzed:
  • Structural Quality: AlphaFold2 pLDDT scores at mutation sites.
  • Structural Differences: Root Mean Square Deviation (RMSD) between the AlphaFold models used for the two versions.
  • Variant Types: Specific amino acid substitutions (e.g., to aromatic residues or proline) driving discrepancies.

3. Key Results

A. High Global Agreement

• Overall Correlation: The dataset showed strong agreement with a global Pearson correlation of 0.956 and a Mean Absolute Error of 0.52 kcal/mol.
• Per-Protein Statistics: The mean Pearson correlation across individual proteins was 0.933 ($\pm 0.077$), and the mean Cohen's $\kappa$ was 0.814 ($\pm 0.092$).
• Bias: There was no systematic inter-version bias. Mean shifts in $\Delta\Delta G$ were minimal (ranging from -0.8 to -0.04 kcal/mol), indicating that FoldX5.1 did not globally recalibrate the score scale.

B. Identification of Outliers

Only three proteins (NUPR1, TSC1, and TMEM127) exhibited poor agreement ($r < 0.7$ or $\kappa < 0.5$).

• NUPR1 & TSC1: Disagreements were strongly enriched in regions with low AlphaFold2 confidence (pLDDT).
  • NUPR1: Mean pLDDT dropped from 75.7 (all sites) to 61.8 (disagreement sites).
  • TSC1: Mean pLDDT dropped from 66.2 to 51.8.
  • Conclusion: Discrepancies here are likely due to structural uncertainty in the input models rather than force-field errors.
• TMEM127: Disagreements occurred in high-confidence regions.
  • Conclusion: As a transmembrane protein, TMEM127 highlights the known limitations of general structure-based predictors for membrane proteins. Additionally, the two versions used slightly different AlphaFold models (RMSD 0.41 Å), though the structures were largely superimposable.

C. Specific Variant Patterns

Discrepancies were often driven by specific substitution types:

• Aromatic Residues: FoldX5.1 often predicted neutral effects where FoldX5 predicted destabilization for substitutions to aromatic residues. This aligns with FoldX5.1's updated $\pi$-stacking and aromatic interaction terms.
• Proline & Glycine: Some proline and glycine mutations showed shifts, likely due to updated hydrogen-bonding, helix-related terms, or buried volume corrections in FoldX5.1.
• NUPR1 Specifics: The NUPR1 model used for FoldX5.1 adopted a different conformation (RMSD 3.78 Å) compared to FoldX5, complicating the separation of force-field effects from structural model errors.

4. Key Contributions

1. Validation of FoldX5.1: The study provides empirical evidence that FoldX5.1 is a robust replacement for FoldX5 in large-scale stability assessments, with high concordance across >500,000 variants.
2. Root Cause Analysis of Discrepancies: The authors demonstrated that most disagreements are not due to systematic algorithmic failures but are concentrated in structurally uncertain regions (low pLDDT) or specific membrane protein contexts.
3. Strategic Migration Protocol: The paper proposes and validates a phased transition strategy for database curation, avoiding the need for a full, immediate re-computation of legacy data.
4. Metadata Enhancement: The authors introduced a new metadata annotation in MAVISp to explicitly record the FoldX software version used for each entry, ensuring provenance and transparency.

5. Significance and Implications

• Database Continuity vs. Innovation: The study resolves the tension between maintaining historical data consistency and adopting improved algorithms. It proves that a "staged transition" (keeping legacy data until annual updates, while using the new tool for new/updated entries) is scientifically sound.
• Resource Efficiency: By avoiding a full database rebuild, the MAVISp team can integrate methodological improvements immediately without delaying updates for months or years.
• User Guidance: The findings highlight that while FoldX5.1 is generally superior, users must remain cautious when interpreting stability predictions for proteins with low-confidence AlphaFold models or transmembrane domains, as these remain challenging for current force fields regardless of the version.
• Future-Proofing: The inclusion of software version metadata sets a standard for reproducibility in computational biology databases, allowing users to filter or re-evaluate data based on the specific tool version used.

Conclusion: The authors conclude that FoldX5.1 should be adopted for future MAVISp data collection. The transition will be managed by retaining FoldX5 annotations for existing entries until their scheduled annual update, while all new or revised entries will utilize FoldX5.1, with explicit versioning to ensure transparency.