💻 bioinformatics

A leakage-controlled benchmark shows apparent codon-language-model advantages in synonymous-variant prediction are evaluation artifacts

This paper demonstrates that previously reported advantages of codon language models in synonymous-variant prediction are evaluation artifacts caused by data leakage, as these gains disappear under strict leakage-controlled benchmarks like the newly released CodonBench.

Original authors: Liang, Y., Zhu, W., Liang, H., Pan, X.

Published 2026-08-14

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Liang, Y., Zhu, W., Liang, H., Pan, X.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are trying to teach a robot to understand the secret language of life. In biology, DNA is written in a code made of four letters (A, C, G, T), which are grouped into triplets called "codons." These codons tell the cell which building blocks (amino acids) to use to build proteins. Here is the tricky part: nature is a bit economical with its grammar. There are 64 possible codon triplets, but they only need to make 20 different building blocks. This means many different codons can spell out the exact same building block. It's like having three different words for "cat"—"feline," "kitty," and "puss"—all meaning the same thing.

For a long time, scientists thought these different words for the same thing were just random noise. But recently, a new wave of AI models called "codon language models" started claiming they could read the secret meaning hidden in these word choices. They argued that the specific word chosen (the codon) changes how stable the message is or how fast the protein is built, even if the final protein is identical. This is a big deal because it could help us design better medicines and vaccines. However, there was a nagging doubt: were these AI models actually reading the secret language, or were they just relying on shortcuts by memorizing the wrong clues?

This paper is a detective story that investigates exactly that question. The authors, Yuanqing Liang, Weimin Liang, and their team, set out to test if these AI models really have a superpower for reading codon choices, or if their success was just an illusion created by a flawed test. They built a new, stricter testing ground called CodonBench to see what happens when you remove the shortcuts.

The Great Shortcut Scandal

The story begins with a confusing discovery. In previous studies, codon-based AI models seemed to crush their competitors. When asked to predict if a specific change in the DNA was harmful, these models were often 2.3 to 14.3 percentage points more accurate than models that only looked at the final protein. It looked like a massive victory for the codon models.

But the authors suspected a trick. Imagine you are taking a test where you have to guess if a student is good at math. If you are allowed to look at the student's name, you might guess "Yes" because you know that specific student is a math whiz, not because you actually looked at their test answers. In the world of DNA, the "name" is the gene.

The authors realized that the old tests were like letting the AI see the student's name. They split the DNA data randomly, meaning the same gene (the same "student") appeared in both the training set (where the AI learned) and the test set (where it was graded). The AI wasn't learning the subtle rules of codon choices; it was just memorizing that "Gene X usually has bad variants" or "Gene Y usually has good ones." It was a shortcut, not a skill.

The "Gene-Held-Out" Trap

To catch the shortcuts, the authors introduced a strict rule: Gene-Held-Out Evaluation. This is like taking a test where you are given a student you have never seen before. You can't use their name to guess; you have to actually read their answers.

When they applied this strict rule, the magic disappeared.

The huge advantage of the codon models collapsed.
The gap between codon models and protein models shrank from a massive 2.3–14.3 percentage points down to a tiny 1.6–2.2 percentage points.
In some cases, the codon models actually got worse than the protein models!

The authors found that the "superpower" was actually just a memory trick. The AI had learned to recognize the gene family rather than the specific biology of the codon.

The "Depth" Illusion

There was one last clue that seemed to prove the codon models were real. The researchers noticed that when they made the AI "think harder" by using a more complex brain (a nonlinear probe), the codon models got even better. This was called the "depth signature." It looked like the AI was digging deeper to find the secret signal.

But the authors dug deeper themselves. They realized this "depth signature" was also a trick. It turned out that the specific software tools used to build the test (like using a specific random number generator or a specific way of organizing the data) were accidentally helping the AI. When they stripped away these software quirks and used a clean, standard setup, the "depth signature" vanished. The extra points the AI gained weren't from understanding biology; they were from the test itself being slightly biased.

The "Best-Epoch" Shortcut

The investigation didn't stop there. The authors found a second, sneaky way the tests were rigged, specifically when the AI was asked to predict numbers (like how much protein a gene would make) rather than just "good" or "bad."

In these number-prediction tasks, the old tests let the AI peek at the final answers while it was still learning. Imagine a student taking a practice exam, but every time they get a question wrong, the teacher whispers the correct answer before they move to the next question. The student then picks the version of their brain that got the highest score on that practice exam and claims they are a genius.

The authors called this "Best-Epoch Selection Bias." The AI was allowed to look at the test set to decide which version of itself was the "best." When they stopped this shortcut by forcing the AI to pick its best version based on a hidden practice set (validation set) instead of the real test, the massive gains vanished. In fact, for some tasks, the AI's performance actually dropped significantly, proving that the previous "success" was just the AI memorizing the test answers.

The Final Verdict: No Magic, Just Noise

The authors tried one last trick to see if there was any truth left. They scrambled the codons randomly—keeping the protein the same but changing the "words" used to spell it. If the AI was really reading the secret language, scrambling the words should make its score drop.

At first, it looked like the score dropped. But when the authors looked at the data with a magnifying glass (using pooled statistics), they realized the drop was just random noise. The difference was so small (0.3 percentage points) that it was statistically meaningless. It was like flipping a coin and thinking you found a pattern because you got three heads in a row.

What This Means

The paper concludes that the apparent advantage of codon language models in predicting harmful DNA changes is an artifact, a mirage created by how the tests were set up.

The Claim: Codon models are better at reading the secret language of DNA.
The Reality: Under strict, leak-free testing, they are not. The signal they were supposedly reading is so faint (about 0.04 bits of information) that current AI models and data sizes can't reliably find it above the noise.

The authors didn't say the secret language doesn't exist; they just said our current tools are too noisy to hear it. They built CodonBench, a new, fair testing ground that prevents these shortcut methods (like gene-memorization and peeking at test answers), so future scientists can stop chasing ghosts and start looking for the real signal.

In short, the AI models weren't smarter than we thought; the tests were just too easy. Now that the tests are harder, the models have to prove they can actually read the code, not just memorize the names or the answers.

Technical Summary: Apparent codon-language-model advantages in synonymous-variant prediction are evaluation artifacts

Problem Statement
Synonymous codon choices influence mRNA stability, translation, and folding, leading to the hypothesis that codon language models (cLMs) should outperform protein language models (pLMs) on tasks involving synonymous variants. However, current evaluations often report large advantages for cLMs without controlling for data leakage or protocol-specific artifacts. The authors argue that when the true biological signal in the synonymous channel is thin relative to confounding factors (such as gene identity), standard evaluation protocols do not measure the signal but rather manufacture the reported gains.

Methodology: CodonBench
The authors introduce CodonBench, a leakage-controlled benchmark designed to isolate the synonymous channel signal from confounding variables. The framework is built on an information-theoretic decomposition: $I(CDS; Y) = I(A; Y) + I(\sigma; Y | A)$ , where $I(A; Y)$ is the amino-acid channel and $I(\sigma; Y | A)$ is the synonymous channel.

The evaluation framework operates across five dimensions:

Data Split: Contrasts standard random splits (which allow gene identity leakage) with Leave-One-Gene-Out Cross-Validation (LOGO-CV), where all variants of a gene are held out from training.
Probe Depth: A capacity-graded ladder ranging from zero-shot likelihood (Level 0) to linear probes (Level 1), nonlinear MLPs (Level 2), and LoRA fine-tuning (Level 3).
Epoch Selection: Contrasts "best-epoch" selection (using test-set performance to select the epoch) with validation-based early stopping.
Pooling Strategy: Compares CLS token pooling (common in cLMs) against mean pooling.
Probe Implementation: Systematically compares different MLP implementations (e.g., scikit-learn vs. PyTorch) and configuration choices (random seeds, weight decay, early stopping).

The study evaluates 21 targets, including 15 neural models (9 cLMs, 2 character-level CDS LMs, 2 pLMs, 2 DNA LMs) and 6 non-neural baselines. A critical diagnostic tool is the onehot_pos canary, a memorization baseline that encodes only (position, codon) identity without learned representations.

Key Results

The "Codon Advantage" is Protocol-Dependent: Under standard random splits, cLMs show large advantages over pLMs on synonymous tasks (tokenization gaps of +2.3–14.3 percentage points; pretrained codon leads of +2.9 pp over ESM-1b). However, these gains vanish or reverse under LOGO-CV. The tokenization gap collapses to +1.6–2.2 pp, and for the largest model (v4), it reverses to favor character tokenization.
Gene-Identity Leakage is the Primary Driver: The onehot_pos memorization baseline outperforms every neural model under random splits (AUC 0.891) but drops significantly under LOGO-CV (AUC 0.739). This indicates that standard protocols allow models to rely on gene identity rather than reading synonymous biology. The collapse is channel-specific: SynPath performance drops 12.7–23.6 pp under LOGO-CV, while MisPath drops only 5.3–7.8 pp, consistent with the thinness of the synonymous signal.
Probe Implementation Artifacts: A single surviving residual under LOGO-CV (a +4.7 pp gain for CodonBERT using an MLP) was dissected via a single-variable ablation (M0–M5). The gain was found to be the cumulative product of six defensible defaults: fixing the random seed (+1.5 pp), early stopping (+0.2 pp), weight decay (+0.6 pp), and switching to the scikit-learn implementation (+3.0 pp). No single choice was erroneous, but their combination fabricated the signal.
Regression Selection Bias: In regression tasks, apparent nonlinear gains (MLP over Ridge) were shown to be artifacts of selecting the best epoch based on test-set performance. When using validation-based early stopping, these gains collapsed or reversed (e.g., from +23.7 pp to -33.0 pp).
Synonym Randomization: A synonym-randomization intervention, which initially appeared to confirm a residual codon signal (5.2 pp drop), was shown to be variance under pooled, paired analysis (0.3 pp, $p=0.49$ ). Furthermore, the randomized model sometimes performed worse than a linear probe on the same features, indicating the "drop" was optimization noise rather than information loss.

Significance and Claims
The paper claims that the reported advantages of codon language models in synonymous-variant prediction are evaluation artifacts, not properties of the models. The authors assert that:

Standard protocols manufacture gains: The combination of random splits, best-epoch selection, and specific probe implementations creates a "leakage cascade" that inflates performance on thin-signal channels.
The signal is likely too thin to detect: The true synonymous channel signal is estimated at $I(\sigma; Y | A) \approx 0.04$ bits. At current sample sizes ( $n=2,840$ ) and with frozen embeddings, this signal is below the noise floor of fold variance and is not reliably detectable.
Methodological Transferability: The "anomaly-control cascade" used in this study (where each anomaly forces a new control) serves as a template for auditing any thin-channel representation-learning claim.
CodonBench as a Standard: The authors release CodonBench to make leakage-controlled evaluation the "path of least resistance," featuring automated pre-flight checks for gene identity leakage and epoch selection bias.

The authors conclude that while a thin synonymous signal may exist, current reports of codon-model superiority are not robust. They caution against reporting advantages from random-split or single-run evaluations and recommend re-examining existing literature under leakage-controlled protocols.