Protein language models learn the underlying accessibility of mutations alongside protein fitness landscapes

Technical Summary: Protein Language Models Learn the Underlying Accessibility of Mutations Alongside Protein Fitness Landscapes

Problem Statement
Protein language models (PLMs), such as ESM-2 and ESM-C, are increasingly used to map protein fitness landscapes by scoring amino acid replacements as pseudo-probabilities. These scores are interpreted as proxies for structural and functional constraints imposed by natural selection. However, a critical confounding factor exists: PLMs are trained on natural amino acid sequences, which are the product of both protein fitness and the underlying biases of nucleotide mutation processes.

Evolutionary trajectories are not driven solely by fitness; they are heavily constrained by mutational accessibility. Certain amino acid replacements are orders of magnitude more likely to occur than others due to:

1. Directional mutation biases (e.g., host-driven A->G and T->C biases in influenza).
2. Codon structure: Replacements requiring one nucleotide change are vastly more accessible than those requiring two or three.
3. Synonymous codon usage: Direct selection at the nucleotide level can skew codon usage.

The paper posits that PLMs implicitly learn these mutational constraints, conflating them with protein fitness. This conflation limits the utility of PLMs for both evolutionary forecasting (predicting which variants will emerge) and synthetic design (isolating pure fitness effects).

Methodology
The authors utilized the rapid emergence of the divergent influenza A H3N2 K lineage (defined by nine HA mutations relative to the J.2 ancestor) as a stress test. They employed a comparative framework involving:

• Models: Base PLMs (ESM-2 650M, ESM-C 600M) and fine-tuned versions (HA-80 for Influenza, fine-tuned ESM-C for SARS-CoV-2).
• Mutational Accessibility Metrics: Empirical nucleotide mutation probabilities derived from Pauly et al. (2017) for H3N2 and De Maio et al. (2021) for SARS-CoV-2. These were used to calculate the probability of specific amino acid transitions based on codon-to-codon mutation rates.
• Information Compression Analysis: A pipeline was designed to compress empirical 64×64 codon mutation matrices into 20×20 amino acid matrices to quantify how much mutational information is theoretically recoverable from amino acid sequences alone.
• Parameter Sweep ($\alpha$): To disentangle fitness from accessibility, the authors introduced a weighting parameter $\alpha$ in the scoring function: $Score = PLM_{prob} \times Mutation_{prob}^\alpha$.
  • $\alpha = 0$: Raw PLM probability.
  • $\alpha = 1$: Explicit coupling of PLM probability with mutational accessibility.
  • $\alpha < 0$: Down-weighting accessible mutations to isolate fitness.
• Evaluation Tasks:
  1. Binary Prediction: Predicting which amino acid replacements are observed in circulating lineages.
  2. Frequency Prediction: Predicting the frequency of observed variants (top 5% of allele frequencies).
  3. Deep Mutational Scanning (DMS): Correlating PLM scores with experimental fitness data (cell entry, pH stability, sera escape, preference ratio) for Influenza and SARS-CoV-2.
  4. Codon Permutation: Testing if PLMs distinguish between identical amino acid changes based on the mutational distance of the starting codon.

Key Results

1. Implicit Learning of Mutational Bias: Both base and fine-tuned PLMs implicitly learn mutational accessibility. While raw probabilities show weak correlation with mutational supply, log-scale correlations are strong ($r \approx 0.63$ for fine-tuned HA-80). Fine-tuning on specific viral sequences significantly amplifies this bias, allowing models to learn specific codon usage patterns and nucleotide mutation matrices without explicit codon input.
2. Impact on Evolutionary Forecasting:
   • Emergence Prediction: Explicitly incorporating mutational accessibility ($\alpha > 0$) significantly improves the binary prediction of which amino acid replacements will be observed (AUROC increased from 0.928 to 0.950 for HA-80). This confirms that mutational supply is a primary driver of which variants emerge.
   • Frequency Prediction: Conversely, when predicting the frequency of already circulating variants, adding mutational accessibility degrades performance. This suggests that once a variant has overcome the mutational barrier, selection dominates the dynamics, and the mutational signal becomes a confounding variable.
3. Impact on Fitness Prediction (DMS):
   • For Deep Mutational Scanning data, the best correlation between PLM scores and experimental fitness was achieved when mutational accessibility was down-weighted ($\alpha < 0$).
   • Fine-tuned models showed the strongest improvement when their scores were inversely weighted by accessibility, indicating that the implicit learning of mutational bias in fine-tuned models actively obscures the underlying protein fitness signal.
4. Codon Distance Sensitivity: PLMs distinguish between amino acid changes based on the number of nucleotide mutations required (1nt vs. 2nt vs. 3nt). Fine-tuned models show a significant preference for substitutions originating from "easy" (1nt) codons, even when the biochemical change is identical, confirming they have learned the topology of codon space.

Significance and Claims
The paper claims that current PLM pseudo-probabilities are composite metrics that conflate protein structural fitness with historical biases in mutational supply. The authors argue that:

• Fine-tuning exacerbates bias: While fine-tuning improves general predictive power for evolutionary tasks, it deeply embeds organism-specific mutational biases into the model, which can be detrimental for isolating pure fitness effects.
• Decoupling is necessary: To optimize predictive accuracy for real-world pathogen forecasting and synthetic design, researchers must explicitly decouple these independent evolutionary processes.
  • For forecasting emergence, models should combine predicted fitness with explicit mutational accessibility.
  • For synthetic design or fitness mapping, models should down-weight mutational accessibility to isolate structural fitness constraints.
• Limitations of current models: The study demonstrates that PLMs trained on amino acid sequences alone can infer complex nucleotide-level constraints, but this capability acts as a confounder when the goal is to understand the fitness landscape in isolation.

The authors conclude that separating mutational supply from structural selection provides a clearer framework for mapping evolutionary trajectories, acknowledging that viral evolution is stochastic and that current models cannot fully predict all trajectories without accounting for these distinct forces.