ViroGym: Realistic Large-Scale Benchmarks for Evaluating Viral Proteins

The paper introduces ViroGym, a comprehensive benchmark comprising extensive deep mutational scanning data and real-world tasks to evaluate protein language models for predicting viral variant effects and guiding rational antigen selection for vaccine development.

The Gist

Imagine you are trying to predict the weather for next year to decide what kind of seeds to plant. You have two main tools:

1. The Lab Test: You take a tiny sample of soil, put it in a controlled environment, and see how it reacts to rain and sun.
2. The Weather Model: You use a super-computer that has read every weather report from the last 1,000 years to guess what the future holds.

For decades, scientists have relied mostly on the Lab Test to figure out how viruses (like the flu or SARS-CoV-2) will change. They grow viruses in a dish, mutate them, and see which ones survive. This is called Deep Mutational Scanning (DMS). It's accurate, but it's slow, expensive, and only tests a tiny fraction of possibilities.

Then came Protein Language Models (pLMs). Think of these as "Google Translate for biology." They have read millions of protein sequences (the "words" of life) and learned the "grammar" of how viruses evolve. They can guess how a virus might change without ever seeing it in a lab.

The Problem:
Until now, we didn't have a good way to test if these "Google Translate" models actually work for viruses. Most of them were trained on human or animal proteins, and the creators often deliberately removed viral data from their training to avoid bias. So, we didn't know if they could really predict the next big pandemic strain.

The Solution: ViroGym
The authors of this paper built ViroGym. Think of ViroGym as a massive, high-stakes gymnasium for AI models. It's a giant playground where they put different AI models through three specific "workouts" to see which one is the strongest athlete for predicting viral evolution.

Here is how the three workouts work, using simple analogies:

Workout 1: The "Mutation Gym" (Mutational Effect Prediction)

• The Task: The AI is shown a virus and asked, "If we change this one letter in the virus's code, will it get stronger or weaker?"
• The Analogy: Imagine a mechanic trying to guess what happens if you swap a specific bolt in a car engine. Some bolts are critical (the car stops), some are useless (nothing happens), and some make the car faster.
• The Result: The AI models were surprisingly good at this. They could look at a mutation and guess its "fitness" (how well the virus survives) almost as well as the slow, expensive lab tests.

Workout 2: The "Disguise Test" (Antigenic Diversity)

• The Task: Viruses are like master thieves; they wear masks (antigens) to hide from our immune system (the police). This workout asks: "If the virus changes its mask, will our current vaccines (the police) still recognize it?"
• The Analogy: Imagine a "Wanted" poster. The AI has to look at a new criminal's face and guess, "Will the police recognize this guy, or has he changed his appearance enough to escape?"
• The Result: This was the hardest workout. The AI models were okay, but not perfect. They struggled to predict exactly how well a vaccine would work against a new strain. There is still a lot of room for improvement here.

Workout 3: The "Crystal Ball" (Pandemic Prediction)

• The Task: This is the ultimate test. The AI looks at the virus today and tries to predict which mutations will become the "dominant" ones in the real world tomorrow.
• The Analogy: Imagine a sports analyst trying to predict which player will become the MVP of the league next season. They have to ignore the noise and pick the players who will actually succeed in the real game, not just in practice.
• The Big Surprise:
  • The Lab Tests (DMS) were great at finding mutations that worked in the dish, but they missed many of the mutations that actually took over in the real world. It's like a player who is great in practice but freezes during the big game.
  • The AI Models (specifically one called ProGen2) were the winners. They predicted the mutations that actually became dominant in the real world (like the famous N501Y mutation in SARS-CoV-2) much better than the lab tests did.

Why This Matters

The paper concludes that AI models are better "crystal balls" than we thought.

While lab tests are still essential for understanding how a virus works in a controlled setting, they might be too narrow to predict how a virus will evolve in the messy, chaotic real world. The AI models, having "read" the history of evolution, seem to understand the bigger picture of what makes a virus successful.

The Takeaway:
ViroGym proves that we can use these AI models to help design vaccines before the virus even becomes a major threat. Instead of waiting for the virus to mutate and then scrambling to make a new vaccine, we can use these "Protein Language Models" to guess the next move, giving us a head start in the race against evolving diseases.

In short: The AI isn't just a translator anymore; it's becoming a fortune teller for viruses.

Technical Summary

Here is a detailed technical summary of the paper "ViroGym: Realistic Large-Scale Benchmarks for Evaluating Viral Proteins."

1. Problem Statement

Protein Language Models (pLMs) have demonstrated significant potential in predicting the functional effects of missense variants in zero-shot settings. However, their application to viral proteins remains under-evaluated and lacks systematic benchmarks.

• The Gap: Current benchmarks (e.g., ProteinGym) are dominated by non-viral proteins. Existing viral benchmarks (e.g., EVEREST) are limited in scope and often fail to integrate in silico predictions with in vitro validation or real-world epidemiological data.
• The Challenge: Viral evolution (e.g., Influenza, SARS-CoV-2) occurs rapidly, often outpacing vaccine development. Current vaccine strain selection relies on semi-annual WHO recommendations, which frequently result in mismatches between predicted and actual circulating strains, leading to suboptimal vaccine efficacy (e.g., 19–60% for seasonal flu).
• The Need: There is a critical need for a comprehensive framework to evaluate pLMs on viral sequences to enable proactive vaccine design, antigen selection, and pandemic forecasting without relying on prior evolutionary or epidemiological data.

2. Methodology: The ViroGym Framework

The authors introduce ViroGym, a large-scale benchmark designed to evaluate pLMs in zero-shot settings across three core tasks. The framework integrates three distinct data sources:

A. Dataset Composition

ViroGym aggregates 79 Deep Mutational Scanning (DMS) assays, 21 influenza neutralization assays, and real-world genomic surveillance data.

• Scale: 552,937 mutated amino acid sequences, 2,691 viral sequence–titer pairs, and 24,187 naturally occurring mutation frequency measurements.
• Coverage: 13 virus types (including SARS-CoV-2, Influenza A, HIV, Zika, Rabies, etc.) and 7 phenotypic categories (binding, cell entry, expression, fitness, immune escape, stability, viral growth).
• Sources:
  • DMS: High-throughput experimental data mapping genotypes to phenotypes.
  • Neutralization Assays: Sequence-based high-throughput assays measuring antibody neutralization titers across viral strains.
  • GISAID: Real-world genomic data for SARS-CoV-2 (Jan 2020 – May 2025) to track natural viral evolution.

B. Evaluation Tasks

1. Mutational Effect Prediction: Evaluates the model's ability to infer functional consequences of single mutations (fitness landscapes).
2. Antigenic Diversity Prediction: Assesses the model's capacity to predict immune escape and strain differentiation (crucial for vaccine strain selection).
3. Pandemic Prediction: A zero-shot generalization task where models predict dominant circulating mutations using only the target protein sequence, validated against real-world GISAID data.

C. Baselines and Scoring Strategies

The study benchmarks various pLMs, including:

• Encoder-only: ESM-1, ESM-1v, ESM-2 (various sizes), ProtT5.
• Decoder-only/Generative: ProGen2 suite, ProtGPT2, Tranception.
• Scoring Strategies: The paper compares methods such as masked marginals, wild-type marginals, mutation marginals, pseudo-likelihood, perplexity, and semantic distance (Euclidean distance between contextual embeddings).

3. Key Contributions

• First Unified Viral Benchmark: ViroGym is the first benchmark to systematically integrate in vitro experimental data (DMS/Neutralization) with in vivo real-world evolutionary data (GISAID) specifically for viral proteins.
• Introduction of "Immune Escape" Phenotype: Unlike previous benchmarks, ViroGym explicitly includes immune escape assays, a critical phenotype for vaccine and therapeutic development.
• Discovery of Scoring Superiority: The authors demonstrate that for encoder-based models, semantic scoring (calculating Euclidean distance between the contextual embeddings of wild-type and variant sequences) outperforms traditional likelihood-based metrics (like masked marginals) in predicting mutational effects.
• Indirect Validation Insight: The study reveals that models selected based on DMS performance (controlled lab conditions) are surprisingly effective at predicting real-world circulating variants, suggesting DMS serves as a useful proxy for identifying models that capture generalizable biological constraints.

4. Key Results

• Mutational Effect Prediction:
  • ProGen2-XL achieved the strongest performance across the DMS benchmark (Average Top 10% Recall: 0.198, Spearman Correlation: 0.293).
  • Semantic scoring strategies significantly outperformed other methods for encoder models (e.g., ESM-2 650M with semantic scoring achieved a Spearman of 0.169 vs. 0.109 for masked marginals).
• Antigenic Diversity Prediction:
  • Performance across models was marginal, with Tranception M slightly outperforming others (Spearman: 0.231). This indicates a significant room for improvement in predicting fine-grained antigenic distances.
• Pandemic Prediction (Real-World Generalization):
  • ProGen2-XL dominated this task, achieving a Precision@3 of 0.33 and a Spearman correlation of 0.315 with GISAID mutation frequencies.
  • Crucial Finding: ProGen2-XL's top predictions overlapped ~50% with dominant mutations observed in real-world GISAID data. In contrast, DMS assays alone only overlapped ~10% with these real-world dominant mutations.
  • The model successfully identified key mutations like N501Y, a major determinant of SARS-CoV-2 Alpha variant transmissibility.

5. Significance and Implications

• Shift in Evaluation Paradigm: The paper argues that pLMs should not merely be evaluated on their ability to reproduce in vitro DMS results. Instead, their true value lies in capturing real-world mutagenic patterns and evolutionary constraints that govern viral spread.
• Complementary Tools: DMS assays and pLMs provide complementary signals. DMS offers high-resolution functional data under controlled conditions, while pLMs capture broader evolutionary constraints learned from massive datasets. Combining both offers a robust strategy for forecasting viral evolution.
• Vaccine Design: The framework provides a pathway for proactive vaccine design. By identifying models that accurately predict dominant circulating mutations (like ProGen2-XL), scientists can initiate manufacturing preparations before official WHO strain announcements, potentially mitigating the lag time that currently reduces vaccine efficacy.
• Future Directions: The authors highlight limitations, such as the difficulty of handling insertions/deletions (indels) and context window limits for long viral proteins, suggesting future work should focus on expanding token representations and context lengths.

In summary, ViroGym establishes a rigorous, multi-faceted benchmark that validates the utility of protein language models in real-world viral surveillance, demonstrating that specific models (notably ProGen2-XL) can effectively bridge the gap between controlled laboratory experiments and the complex dynamics of natural viral evolution.