🧬 biology

Viral Proteins Reveal Geometry of Protein Language Models

This study demonstrates that while protein language models organize viral and cellular sequences along a dominant "nativeness" axis based on reconstruction perplexity, their embeddings still retain sufficient viral-specific signals to allow for linear separability beyond simple zero-shot metrics.

Original authors: Arthur Bigot, Harmon Bhasin, Core Francisco Park, Eugene Shakhnovich, Dianzhuo Wang

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Arthur Bigot, Harmon Bhasin, Core Francisco Park, Eugene Shakhnovich, Dianzhuo Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The Big Picture: AI Learning the "Language" of Life

Imagine you have a super-smart AI that has read almost every book ever written about biology. This AI is a Protein Language Model (pLM). Just as a human learns English by reading millions of books, this AI learns the "language" of proteins by reading millions of genetic sequences.

The researchers wanted to know: If this AI reads mostly books about humans, plants, and bacteria, does it understand the "dialect" spoken by viruses?

Viruses are tricky. They are like a small, noisy minority in a huge library. They mutate fast, have tiny genomes, and rely on their hosts to survive. The study found that the AI does understand them, but in a very specific, geometric way.

1. The "Native" vs. "Foreign" Axis

The researchers discovered that inside the AI's brain, there is a giant invisible ruler (a mathematical line) that sorts all proteins. Let's call this the "Nativeness Scale."

The Left Side (The "Home Team"): On one end of the ruler sit proteins from humans, bacteria, and plants. These are the "native" proteins. The AI has read about them a billion times. They fit perfectly into the AI's expectations.
The Middle (The "Guests"): In the middle of the ruler sit viral proteins. They aren't "wrong," but they feel a bit like guests who don't quite know the local customs. They are distinct from the home team but still make sense.
The Right Side (The "Gibberish"): On the far end of the ruler are random, scrambled strings of amino acids. These are like sentences with words in the wrong order or made-up words. The AI thinks these are nonsense.

The Discovery: The AI doesn't just see viruses as "bad" or "random." It sees them as a specific, organized group that sits between the "perfectly normal" cellular proteins and "total nonsense."

2. The "Confusion Score" (Perplexity)

How does the AI know where to put a protein on this ruler? It uses a score called Perplexity.

Think of it like a game of "Guess the Next Word." If you say, "The cat sat on the...", the AI is very confident the next word is "mat." It has low confusion (low perplexity).
If you say, "The cat sat on the...", and the next word is "banana," the AI is very confused (high perplexity).

The study found that cellular proteins are like "mat" (easy to guess). Viral proteins are like "banana" (a bit harder to guess, but still a real word). Random junk is like "flibber" (total nonsense).

The AI's internal ruler is almost perfectly aligned with this "Confusion Score." The more confused the AI is, the further to the right the protein sits on the ruler.

3. Making the AI Bigger Doesn't Fix Everything

Usually, when you make an AI bigger (give it more brain power and data), it gets better at everything. The researchers tested this by making the AI models larger and larger.

The Result: The bigger models did get better at understanding viruses, but not equally for all viruses.
The Analogy: Imagine a teacher trying to learn a new accent.
- Some viral families (like Retroviridae) are like a student who speaks a dialect very similar to the teacher's native tongue. As the teacher gets smarter, they understand these students almost perfectly.
- Other viral families (like Orthomyxoviridae, which includes the flu) are like a student speaking a completely different, complex dialect. Even when the teacher gets super-smart, they still struggle to understand these specific students.

So, making the AI bigger helps, but it doesn't make all viruses suddenly look "normal." Some remain distinct and "foreign" to the model.

4. The AI Knows More Than Just "Confusion"

Here is the most surprising part. The researchers asked: Is the AI just separating viruses because they are "confusing" (high perplexity), or does it actually understand what makes a virus a virus?

They built a simple test:

Method A: Just use the "Confusion Score" to guess if a protein is viral.
Method B: Use the AI's deep internal "thoughts" (embeddings) to guess.

The Result: Method B (the deep thoughts) was much better than Method A.
Even when the AI got so good at viruses that they stopped looking "confusing" (low perplexity), the AI's internal map still knew exactly which ones were viruses.

The Analogy: Imagine you are trying to spot a spy in a crowd.

Method A is looking for people who look nervous (high confusion).
Method B is looking at their body language, gait, and how they hold their hands (the embedding).
The study found that even when the spy stops looking nervous, the AI can still spot them because it has learned the subtle, specific "viral signature" that goes beyond just being confused.

Summary of Findings

There is a "Nativeness" Line: The AI organizes proteins on a line from "Very Normal" to "Very Weird." Viruses sit in the middle.
Bigger isn't Always Better for Everyone: Bigger AI models help some viruses look more "normal," but others stay distinct.
The AI Knows the Difference: The AI doesn't just separate viruses because they are hard to predict; it actually retains a specific "viral signal" that allows it to identify them even when they look very similar to normal proteins.

Why this matters (according to the paper):
This helps us understand how these powerful AI tools "see" the biological world. It shows that while these models are trained mostly on human and animal data, they have developed a structured way to handle the unique, fast-changing world of viruses. This is crucial for knowing when to trust the AI's predictions and when to be careful.

Technical Summary: Viral Proteins Reveal Geometry of Protein Language Models

Problem Statement
Protein language models (pLMs) are trained on highly imbalanced datasets dominated by cellular organisms, raising questions about how they represent underrepresented biological groups. Viral proteins, which are evolutionarily distinct and shaped by different constraints (e.g., host dependence, high mutation rates), are less abundant in standard pretraining data. While pLMs perform well on standard benchmarks, they struggle with viral mutation-effect prediction. The central questions addressed are: (1) Is the separation of viral proteins in embedding space primarily explained by lower "nativeness" (i.e., they are less well-modeled by the pretraining distribution)? and (2) Beyond this nativeness gap, do embeddings retain specific viral information that is not captured by reconstruction difficulty or shallow sequence features?

Methodology
The authors analyze viral protein representations across three families of pLMs (ESM2, ESMC, and ESM3) spanning over three orders of magnitude in parameter count.

Datasets: The study utilizes a multi-group biological dataset comprising ten groups: six cellular (bacteria, archaea, plants, fungi, insects, human non-viral) and four viral (bacteriophage, human, plant, invertebrate). Controls include position-shuffled cellular/viral sequences and i.i.d. uniform random sequences. A specific human viral/cellular classification dataset (5,203 viral vs. 5,197 cellular sequences) is used for downstream evaluation.
Nativeness Metric: Masked-reconstruction perplexity (PPL) is used as a model-relative measure of nativeness. Lower PPL indicates a sequence is better modeled by the pretraining distribution.
Geometric Analysis: Principal Component Analysis (PCA) is applied to mean-pooled sequence embeddings to identify dominant axes of variance. The alignment between the first principal component (PC1) and PPL is measured.
Scaling Analysis: The fraction of viral sequences falling below a native-like PPL threshold is tracked across model scales to assess how scaling affects different viral families.
Classification Evaluation: To determine if viral signal persists beyond reconstruction difficulty, the authors compare three classifiers on a held-out test set:
1. A linear probe (logistic regression) trained on mean-pooled embeddings.
2. A zero-shot classifier based solely on negated PPL.
3. Shallow baselines based on sequence length, amino acid composition, and dipeptide composition.

Key Contributions and Results

A Dominant Nativeness Axis:
The study identifies a dominant "nativeness axis" (PC1) in pLM representation space that explains a large portion of the variance (e.g., 73.1% in ESMC-600M). This axis is strongly aligned with masked-reconstruction perplexity ( $\rho \approx +0.96$ ). It orders sequences from well-modeled cellular proteins (low PPL) through viral proteins (intermediate PPL) to biologically meaningless shuffled/random sequences (high PPL). This geometric structure is consistent across ESM families and extends to autoregressive (ProGen2) and diffusion (EvoDiff) models, suggesting it is a general property of masked sequence models trained on imbalanced biological data.
Heterogeneous Scaling Effects:
Increasing model scale contracts the nativeness axis, but the effect is uneven across viral families. While the average fraction of "native-like" viral proteins increases with scale, specific families respond differently. For instance, Papillomaviridae and Retroviridae move significantly toward the native region as models scale up, whereas families like Orthomyxoviridae and Orthoherpesviridae remain persistently displaced even at the largest scales (e.g., 6B parameters). The paper suggests this heterogeneity may be linked to the presence of cellular homologs; Retroviridae, which shares domains with eukaryotic retrotransposons, is more compatible with the cellular-dominated prior.
Residual Viral Signal Beyond Nativeness:
Crucially, the study demonstrates that viral identity is linearly encoded in embeddings beyond what is captured by reconstruction difficulty. Linear probes trained on embeddings achieve near-ceiling accuracy (AUC $\in [0.97, 1.00]$ ) across all scales, significantly outperforming both PPL-based zero-shot classifiers and shallow sequence baselines.
- Divergence at Scale: As models scale, PPL-based discrimination often becomes weaker or non-monotonic (dropping for very large models like ESM2-15B) because some viral proteins become easier to reconstruct (lower PPL), reducing the separation available from PPL alone. However, the linear embedding probe maintains high performance, indicating that the embeddings preserve viral-specific structural or functional signals that are not merely artifacts of reconstruction difficulty.
- Robustness: This separability holds even when the negative class is restricted to human host proteins (same host) and in leave-one-family-out cross-validation, confirming the signal is not due to memorizing family-specific motifs or broad taxonomic differences.

Significance and Claims
The paper claims that pLM representations are structured by a general notion of "nativeness" relative to the pretraining prior, which places viral proteins in a displaced but structured intermediate region. However, this displacement does not imply a loss of information; rather, embeddings retain distinct viral signals that are linearly accessible.

The authors suggest that "nativeness" (low PPL) can serve as a diagnostic tool: families with low nativeness may require more cautious zero-shot predictions. Furthermore, the ability of embeddings to distinguish viral from cellular sequences beyond homology-based methods suggests potential utility for biosecurity screening, particularly for detecting engineered or highly diverged variants that standard homology filters might miss. The work positions itself as a contribution to safer biological model evaluation rather than capability deployment, highlighting the need to understand how foundation models represent underrepresented biological groups.