Benchmarking Deep Learning Predictions of Mutation-Induced Fold Switching

Problem Statement
While deep learning models like AlphaFold2 and AlphaFold3 have revolutionized protein structure prediction, their ability to predict the effects of point mutations on conformational dynamics remains unclear. Specifically, it is unknown whether these models can accurately predict mutation-induced fold switching in metamorphic proteins—proteins capable of adopting two distinct, stable, and well-ordered folds. Previous benchmarks suggest AlphaFold captures alternative states in less than a quarter of known fold-switching proteins and struggles to model the structural impact of point mutations. This study addresses the gap in understanding how current deep learning and physics-based methods perform when tasked with predicting how specific residue substitutions alter the equilibrium between folded states and unfolded states.

Methodology
The authors constructed a systematic, NMR-characterized benchmark dataset using the GA/GB model fold-switching system. They performed scanning mutagenesis at three key positions (L45 in GA98, T25 in GB98, and L20 in GB98-T25I), generating all 19 possible amino acid substitutions for each site (60 variants total).

• Experimental Characterization: The fold-state populations (3α, 4β+α, and unfolded) for all 60 wild-type and variant proteins were measured using 2D 1H-15N HSQC NMR spectroscopy.
• Computational Evaluation: The authors evaluated a panel of deep learning and physics-based algorithms against this experimental data. The deep learning suite included AlphaFold2, AlphaFold3, AF-Cluster, AFSample2, CF-random, ESMFold2, ESM3, and Boltz-1x. Inverse folding methods (ProteinMPNN, ESM-IF1, Frame2Seq) and a physics-based Rosetta ΔΔG protocol were also assessed.
• Analysis Metrics: Model outputs (1,000 models per mutant) were classified by TM-align against reference structures. Performance was quantified using RV coefficients (multivariate correlation between predicted and experimental state proportions) and Brier scores (state classification error). Additionally, predictions were correlated with independent MegaScale thermostability (ΔΔG) measurements.
• Extended Systems: The methods were further tested on mutants of the Sa1 and KaiB metamorphic systems to assess generalizability.

Key Contributions

1. New Benchmark Dataset: The study provides a quantitative, NMR-derived benchmark of 60 point mutants across three critical positions in the GA/GB system, detailing their specific effects on 3α, 4β+α, and unfolded state populations.
2. Systematic Algorithm Assessment: The paper offers a comparative evaluation of state-of-the-art deep learning structure predictors, inverse folding tools, and physics-based scoring functions specifically for mutation-induced fold switching.
3. Recycling Step Analysis: The authors investigated the impact of AlphaFold2's recycling steps on fold prediction, revealing that the bias toward the 3α fold is strongest at 0 recycling steps and diminishes as recycling increases.
4. Thermodynamic Correlation: The study correlates model output distributions with experimental thermostability data (ΔΔG) to determine if models capture the magnitude of destabilization caused by mutations.

Results

• Position-Dependent Performance: Predictive performance varied significantly by residue position. For residue 45 (GA98), AlphaFold2 and AFSample2 showed the highest correlation with experimental data (RV coefficients of 0.49 and 0.48, respectively). In contrast, performance for residues 20 and 25 was generally poor, with most methods showing RV coefficients below 0.2.
• Bias and Sensitivity: All methods exhibited a bias toward the 3α fold. However, for residue 45, AlphaFold2 demonstrated a surprising ability to distinguish between mutants that stabilize the 3α fold, the 4β+α fold, or lead to unfolding. Specifically, the proportion of 4β+α models increased for the 45Y mutant, which experimentally switches to the 4β+α state.
• Recycling Dynamics: Without recycling, AlphaFold2 predicted 92.5% of models as 3α regardless of the true state. As recycling steps increased (up to 20), the proportion of 3α models decreased, and the model's ability to differentiate between mutant groups improved, particularly for unfolded mutants.
• Inverse Folding and Physics-Based Methods: Inverse folding methods and Rosetta showed limited discriminatory power. While Rosetta could predict some destabilizing effects for residue 45, it failed to consistently identify disruptive effects for residues 20 and 25. Inverse folding log-probabilities showed some correlation with ΔΔG for specific residue-site combinations but lacked general predictive power across all sites.
• Thermostability Correlation: AlphaFold2's proportion of unfolded models strongly correlated with experimental ΔΔG values for residue 45 mutants (r = 0.89), suggesting the model captures the magnitude of destabilization at this site. This correlation was absent for residues 20 and 25.
• Generalizability: When applied to Sa1 and KaiB systems, most methods (except AFSample2) failed to capture mutation-induced shifts in state equilibrium, predicting exclusively the dominant state or unfolded models.

Significance
The paper concludes that current deep learning models exhibit partial, position-dependent sensitivity to mutation-induced fold switching. While AlphaFold2 demonstrates a limited but significant capacity to capture the effects of point mutations on state preference—particularly at residue 45 of the GA/GB system—it generally struggles with other positions and systems. The authors posit that this sensitivity may stem from the model's learned energy function or training data biases (e.g., the conservation of tyrosine at position 45 in natural GB domains) rather than a fundamental understanding of the complex energy landscapes of metamorphic proteins.

The study highlights that while these tools are not yet fully reliable for designing or predicting complex fold-switching behaviors, they possess a degree of physical insight that warrants further investigation. The authors suggest that future improvements may require explicit incorporation of thermodynamic data, NMR structures (which are underrepresented in training sets), or hybrid approaches combining deep learning with enhanced sampling molecular dynamics. The GA/GB mutant dataset presented serves as a critical, experimentally grounded test set for developing and evaluating such future approaches.