🧬 biology

A permutation-null artifact drives a conservation–divergence spectrum's signature trend, and phylogeny dominates a protein language model's per-position coupling signal

This paper demonstrates that the signature negative trend in the conservation–divergence spectrum is largely an artifact of its permutation null model and that protein language models' per-position coupling signals are predominantly driven by phylogeny rather than functional mechanism, establishing a rigorous framework for cross-examining black-box models against independent evolutionary evidence.

Original authors: Huazhang Shen

Published 2026-09-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Huazhang Shen

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Proteins are the molecular machines that keep life running, and their shapes and functions are dictated by the precise order of their building blocks, called amino acids. For decades, scientists have tried to read these sequences to find the specific spots that determine a protein's unique job. Two very different approaches have emerged to solve this puzzle. The first relies on comparing thousands of related proteins to see which spots stay the same or change together across evolution; this method is transparent and easy to follow, but it can only see patterns written in single letters. The second approach uses artificial intelligence models trained on millions of sequences to predict how a protein behaves; these models are incredibly powerful and can sense complex, hidden patterns, but they are often called "black boxes" because no one is entirely sure what they have actually learned or if they are simply memorizing family trees rather than understanding how the machine works. The question facing the field is whether these intelligent models are discovering new biological truths or just taking shortcuts based on evolutionary history.

A recent study by independent researcher Huazhang Shen brings these two methods face-to-face to test exactly what they know. The researcher focused on a specific group of proteins called G-protein-coupled receptors, which act as switches on the surface of cells, deciding which internal signal to send when they are activated. The goal was to identify the exact spots on these proteins that determine which signal they choose. The study treated the traditional evolutionary method and the artificial intelligence model as two independent observers examining the same set of proteins. One observer looked for specific amino acid changes that distinguish the different signal types, while the other used a neural network to guess the signal type based on the protein's sequence. The researchers then compared the lists of important spots generated by each observer to see if they agreed.

The comparison revealed a surprising lack of agreement. The two methods pointed to almost entirely different spots as being important, with their rankings barely overlapping. This divergence suggested that the two observers were looking at the problem through completely different lenses. To understand why, the researcher first examined the traditional evolutionary method. This method had previously shown a clear pattern: spots that were highly conserved across evolution tended to be less variable between different signal types. While this looked like a biological law, the study found that this pattern was largely an illusion created by the mathematics of the test itself. By shuffling the data to create a statistical baseline, the researcher showed that the negative trend appeared even in random noise. The real biological signal was not the trend itself, but the small amount of extra variation that remained after subtracting this mathematical baseline. This finding implies that many previous studies claiming to find specificity based on this trend might have been measuring a statistical artifact rather than a biological rule.

The investigation then turned to the artificial intelligence model to see what it was actually learning. When the model was tested without any restrictions, it appeared to know a great deal about which signal a protein would choose. However, when the researcher forced the model to prove it understood the mechanism rather than just recognizing evolutionary cousins, the results changed dramatically. By grouping related proteins together during testing so the model could not rely on family resemblance, the model's performance dropped significantly. The study calculated that roughly three-quarters of the model's apparent knowledge was simply a reflection of evolutionary relatedness, not the actual mechanism of how the protein works. Only about one-quarter of its signal represented genuine functional insight. This provides strong quantitative evidence for the criticism that these powerful models often act as shortcuts for recognizing homology rather than true understanding of biological function.

Despite these limitations, the study found a specific area where the artificial intelligence model held an advantage. When the two methods disagreed, the model's predictions were slightly more likely to align with the physical locations where the protein actually touches its internal partner, based on known 3D structures. However, this alignment was weak and appeared as a general shift in the data rather than a perfect list of correct answers. To ensure the methods were working correctly, the researcher also tested them on a different family of proteins where the biological code is known to be too complex for either method to read. In this case, both methods failed to find the correct spots and disagreed with each other, confirming that the framework could distinguish between a genuine biological limit and a failure of the tools.

The study concludes with a new framework for testing what these black-box models actually know. It demonstrates that before trusting a model's predictions, scientists must strip away the influence of evolutionary history and account for statistical baselines that can mimic real patterns. While the artificial intelligence model does capture some real functional information, it is heavily contaminated by family history, and the traditional evolutionary method is prone to misinterpreting statistical noise as biological law. The research does not declare one method a winner but instead provides a rigorous way to cross-examine them, ensuring that future discoveries about how proteins work are based on genuine biological signals rather than mathematical illusions or evolutionary shortcuts.

Technical Summary: A Cross-Examination of Evolutionary and Language-Model Signals in Protein Specificity

Problem Statement
The identification of specificity-determining positions (SDPs) in protein families is a longstanding challenge with two dominant methodological approaches: evolutionary per-position methods (e.g., conservation–divergence spectra, mutual information) and protein language models (PLMs). Evolutionary methods are interpretable but blind to distributed, epistatic signals. PLMs capture distributed context but operate as black boxes, with a persistent critique that their per-position signals reflect phylogenetic relatedness (homology shortcuts) rather than functional mechanism. This paper (P3) addresses the lack of direct, position-by-position comparison between these two approaches and the absence of rigorous null-model discipline required to validate their signals. The study focuses on G-protein-coupled receptor (GPCR) coupling specificity (Gs, Gi-o, Gq-11) as a test case, using opsin spectral tuning as a negative control for "unreadable" biology.

Methodology
The study employs a "two detectives" framework, cross-examining two distinct signals on the same dataset (219 class A GPCRs, 212 structurally equivalent positions):

Physical-Evidence Detective ( $S_{sdp}$ and $S_{net}$ ): Based on the conservation–divergence spectrum from companion preprints (P1, P2).
- $S_{sdp}$ : The anomaly score defined as the between-class Jensen–Shannon divergence (JSD) minus the value predicted by a linear fit of divergence on within-class conservation.
- $S_{net}$ : A refined score where the permutation-null floor is subtracted per position, expressed as a Gamma-fit tail probability to allow cross-family comparison.
- Null Discipline: A permutation null is constructed by shuffling coupling labels while preserving amino-acid composition to determine the expected negative trend between conservation and divergence.
Intuition Detective ( $S_{plm}$ ): A logistic-regression probe trained on ESM-2 embeddings (640-dimensional vectors).
- The probe predicts coupling class from per-position embeddings without prior knowledge of coupling.
- Phylogenetic Subtraction: To isolate mechanism from homology, the study compares random-split cross-validation (total signal) against clade-aware cross-validation (where related receptors are held out together). The difference represents the "pure coupling" signal.

Key Contributions

Null-Model Discipline for Evolutionary Trends: The paper establishes that the signature negative trend between conservation and divergence (often cited as evidence of specificity encoding) is largely a statistical artifact of the permutation null.
Quantitative Decomposition of PLM Signals: It provides a direct, quantitative measure of how much of a PLM's per-position predictive power is due to phylogeny versus functional mechanism.
Cross-Method Validation Framework: It introduces a protocol for auditing black-box models against independent evolutionary evidence and physical gold standards (structural interfaces) at the resolution of individual residues.

Key Results

Divergent Rankings: The two methods largely disagree on which positions are important. The two physical-evidence scores ( $S_{sdp}$ vs. $S_{net}$ ) correlate strongly ( $\rho = 0.88$ ), but the physical evidence and the PLM probe correlate weakly ( $\rho = 0.17$ ).
The Signature Trend is Mostly Null: The observed negative correlation between conservation and divergence ( $\rho \approx -0.61$ ) is mostly the shape of the permutation null ( $\rho_{null} \approx -0.90$ ). In three of four independent families tested (GPCR, opsin, nuclear-receptor DBD), the null explains $\ge 100\%$ of the observed trend. The true signal is the excess above this null, not the trend itself.
Phylogeny Dominates PLM Signals: For the GPCR coupling task, approximately 74% of the PLM's apparent per-position knowledge is explained by phylogenetic relatedness. The "pure coupling" signal (mechanism) accounts for only ~24% of the signal above chance. This validates the critique that PLMs often act as homology shortcuts.
Structural Alignment in Disagreement: Where the methods disagree, the PLM (specifically at layer 30) shows a statistically significant enrichment on the true structural G $\alpha$ coupling interface ( $p=0.003$ ), whereas the physical-evidence method does not ( $p=0.18$ ). However, this is a distribution-level shift, not a "top-list" victory (Fisher exact test on top-20 positions is not significant), and the structural interface is acknowledged as a partial gold standard.
Robustness and Falsification: The dominance of phylogeny in PLM signals holds across ESM layers (6, 15, 21, 30) and probe types. However, a claim that the pure-coupling residual is biased toward conserved positions was found to be an artifact specific to layer 30 and is not a general rule.
Negative Control (Opsin): On opsin spectral tuning (a family established as biologically "unreadable" due to distributed epistasis), both methods fail to recover the known tuning sites in top-list rankings and are uncorrelated ( $\rho = 0.005$ ). This confirms that when biology is unreadable, independent methods fail together.

Significance and Claims
The paper claims to provide a reusable framework for interrogating black-box sequence models. Its primary significance lies in the "null-model discipline" it enforces:

It demonstrates that evolutionary signatures (like the conservation–divergence trend) must be tested against a matched permutation null before being accepted as biological findings.
It quantifies the "homology shortcut" in PLMs, showing that without clade-aware validation, PLM performance significantly overstates mechanistic understanding.
It clarifies that while PLMs may capture structural interface information missed by evolutionary methods, this signal is weak and distributed, not a ranked list of definitive SDPs.

The author explicitly states that the contribution is not a new scoring metric but a methodology for cross-examination. They caution that the "pure coupling" residual is thin and that the structural interface is an incomplete gold standard, meaning the failure of the evolutionary method to enrich on the interface does not prove it is incorrect, only that it may be reading a different (e.g., allosteric) layer of the signal. The paper concludes that the applicability limits of these methods are properties of the biological family (e.g., whether the signal is distributed or local) rather than defects of the detectors themselves.

Technical Summary: A Cross-Examination of Evolutionary and Language-Model Signals in Protein Specificity

More like this