🧬 biology

GENEB: Why Genomic Models Are Hard to Compare

To address the difficulty of comparing genomic foundation models due to fragmented benchmarks and incompatible protocols, the paper introduces GENEB, a unified large-scale diagnostic benchmark that evaluates 40 models across 100 tasks to reveal the instability of aggregate rankings and demonstrate that architectural and pretraining alignment often outweigh model scale.

Original authors: Daria Ledneva, Mikhail Nuridinov, Denis Kuznetsov

Published 2026-08-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Daria Ledneva, Mikhail Nuridinov, Denis Kuznetsov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are trying to teach a computer to read the secret instruction manual of life. This manual is written in a language made of just four letters: A, C, G, and T. These letters form the DNA code that tells every living thing—from a tiny bacterium to a giant oak tree—how to build itself and function. In recent years, scientists have built "foundation models," which are like super-smart students that have read billions of pages of this DNA manual. They are supposed to learn the rules of the language so well that they can predict things like how a gene will behave or if a specific sequence causes a disease.

But here is the problem: imagine if every student in a class took a different test, used a different dictionary, and was graded by a different teacher. One student might be great at spelling but terrible at grammar, while another is a grammar wizard but can't spell. If you just looked at their final grades, you wouldn't know who is actually the best student overall. You wouldn't know if the "grammar wizard" failed the spelling test because they are bad at it, or because the test was unfair. This is exactly what has been happening in the world of DNA AI. Different research teams build their models, test them on their own unique sets of tasks, and claim their model is the "best." But because they aren't taking the same test, nobody can really compare them fairly. It's like trying to decide who is the best athlete in the world by comparing a swimmer's time in a pool to a runner's time on a track without a unified standard.

This is where a new study called GENEB steps in to fix the mess. The researchers, Daria Ledneva, Mikhail Nuridinov, and Denis Kuznetsov, decided to build a massive, unified "Olympics" for DNA models. They gathered 40 different genomic foundation models—some huge, some small, some trained on human DNA, others on bacteria or plants—and put them all through the exact same 100 different challenges. These challenges covered 13 different types of biological tasks, like finding specific switches in the DNA (promoters), identifying how genes are turned on or off (histone modifications), or spotting viruses. They used a strict, fair scoring system to see how each model performed when frozen in place, meaning they couldn't learn anything new during the test; they had to rely entirely on what they had already learned.

The results of this grand experiment were surprising and shook up a few common beliefs. First, the study found that bigger isn't always better. In the world of AI, there's a popular idea that if you just make the model bigger (give it more "brain power" or parameters), it will automatically get smarter. But GENEB showed that this isn't a magic bullet. They found that a tiny model with only 86 million "parameters" (a measure of size) could actually beat a giant model with 7 billion parameters by a huge margin. In fact, the giant model was so bad at some tasks that it performed worse than random guessing, simply because it had been trained on the wrong kind of DNA (bacteria) while the test was about human DNA. It's like taking a chef who is a master at cooking Italian food and asking them to cook a traditional Japanese meal; no matter how famous or expensive the chef is, they will struggle if they don't know the ingredients.

Second, the study discovered that what a model was trained on matters more than how big it is. A model trained specifically on human DNA or a mix of many different species performed much better on human tasks than a massive model trained only on bacteria. The researchers showed that if the "diet" of the model (the data it ate during training) doesn't match the "meal" it's being asked to eat later (the task it's tested on), it fails. For example, models trained on human data were terrible at identifying plant genes, and models trained on bacteria couldn't figure out how human genes are spliced together. The paper suggests that for these models to be truly useful, they need to be fed the right kind of data for the job, rather than just being made larger.

Finally, the paper argues that there is no single "champion" model. Just like in sports, a model might be the best at running but terrible at swimming. The study found that a model that was the top performer on one task could be near the bottom on another. This means that instead of looking for one "best" model to rule them all, scientists and doctors need to pick the right tool for the specific job. If you need to predict how a human gene is regulated, you should pick a model trained on human epigenetic data. If you need to identify plant genes, you need a model trained on plants. The paper concludes that the current way of ranking these models with a single leaderboard is misleading and that we need a more careful, category-by-category approach to choose the right AI for the right biological question.

Technical Summary: GENEB – Why Genomic Models Are Hard to Compare

1. Problem Statement

The rapid expansion of genomic foundation models has outpaced the development of methodological infrastructure for their comparison. Current evaluation practices are characterized by fragmented benchmarks, incompatible protocols, and task-specific reporting. Consequently, claims of model superiority or generality are often not directly comparable; a model may be hailed as a breakthrough in one context while underperforming in another due to the absence of a unified evaluation framework. This fragmentation obscures the true relationships between models and hinders the assessment of genuine progress. Furthermore, the field lacks a systematic understanding of how model scale, architecture, tokenization, and pretraining data interact to influence performance across diverse genomic tasks.

2. Methodology

To address these challenges, the authors introduce GENEB, a large-scale diagnostic benchmark designed to evaluate frozen representations from 40 genomic foundation models across 100 tasks spanning 13 functional categories.

Evaluation Protocol: GENEB employs a unified probing-based protocol. Frozen sequence representations are extracted from each model and evaluated using lightweight logistic regression classifiers. This approach isolates representation quality from the confounding effects of task-specific fine-tuning.
Shot Regimes: Models are evaluated under three regimes: full-data, 10-shot, and 1-shot, allowing for the analysis of few-shot robustness.
Metrics: The primary metric is the Matthews Correlation Coefficient (MCC), chosen for its robustness to class imbalance, which is standard in genomic evaluation.
Benchmark Composition: The 100 tasks are aggregated from nine existing benchmarks (e.g., Nucleotide Transformer, GUE, Genomic Benchmarks) and organized into 13 functional categories, including Histone Modifications, Promoters, Enhancers, DNA Methylation, Splice Sites, lncRNA, and Species Classification. Notably, 12 of the 13 categories focus on eukaryotic genomic phenomena.
Controlled Comparisons: To isolate specific factors, the authors perform matched-pair analyses where models are compared while holding other variables constant (e.g., comparing architectures with the same pretraining corpus and tokenization, or comparing pretraining data with the same architecture and tokenization).

3. Key Contributions

Unified Benchmark: GENEB provides the first exhaustive, controlled evaluation of 40 diverse genomic foundation models on a comprehensive task suite, serving as a reference framework analogous to MTEB in natural language processing.
Diagnostic Framework: By exposing task-level trade-offs and performance heterogeneity, GENEB moves beyond aggregate leaderboards to enable principled, category-aware model selection.
Systematic Analysis: The paper offers a granular dissection of how scale, architecture, tokenization, and pretraining data interact, challenging the assumption that parameter count is the primary driver of performance.

4. Key Results

The Scale–Performance Disconnect

While there is a statistically significant aggregate correlation between model scale and performance ( $\rho = 0.573$ ), scale is an imperfect predictor of category-level outcomes.

In-Domain Reversals: Among in-domain models, there are 25 instances where a model at least 5 $\times$ smaller outperforms a larger counterpart in aggregate MCC.
Domain Mismatch: The correlation strengthens to $\rho = 0.694$ when excluding EVO-1-131K, a 7B-parameter model pretrained exclusively on prokaryotic sequences. This model ranks among the weakest on eukaryotic tasks (e.g., TF binding, splice sites), illustrating that domain mismatch can negate the benefits of massive scale.

Architectural and Pretraining Effects

Architecture: Under controlled conditions (matched pretraining and tokenization), Transformer-based models generally outperform state-space models (e.g., Mamba). However, this is not universal; on Chromatin Accessibility, Mamba-based models (ECCDNAMAMBA) outperform Transformer decoders, suggesting architecture-task alignment is critical.
Pretraining Data: Taxonomic alignment is crucial.
- Multi-species vs. Human-only: Multi-species pretraining generally yields small but consistent gains over human-only pretraining, particularly on Chromatin Accessibility and Splice Sites.
- Eukaryotic vs. Prokaryotic/Microbial: Models pretrained on prokaryotic or microbial-only corpora transfer poorly to eukaryotic tasks. EVO-1-131K (prokaryotic) and DNABERT-S (microbial) significantly underperform eukaryotic-focused models on eukaryotic tasks.
- Gene-Focused vs. General: Gene-focused pretraining (e.g., GENERATOR-EUKARYOTE) shows advantages over broad multi-species pretraining on specific tasks like Splice Sites and lncRNA.

Tokenization

No single tokenization strategy (BPE, k-mer, single-nucleotide) dominates globally. The optimal choice depends on the interaction between architecture and pretraining data. For instance, single-nucleotide tokenization (MUTBERT) outperforms BPE on human-pretrained models for certain tasks, while BPE shows advantages in multi-species decoder settings.

Few-Shot Robustness

Performance degrades sharply in low-data regimes (mean macro-MCC drops from 0.488 in full-shot to 0.106 in 1-shot).

Inverse Pattern: Counterintuitively, models with the lowest full-shot performance often show the smallest absolute drops in few-shot regimes. This is attributed to a low performance ceiling rather than genuine robustness.
Category Sensitivity: Few-shot rankings differ significantly from full-shot rankings in 8 of 13 categories, indicating that the best model for full supervision is not necessarily the best for few-shot deployment.

Task Difficulty and Specialization

Hard Frontiers: Tasks such as DNA methylation prediction and plant lncRNA identification remain largely unsolved, with many models achieving near-random performance. Scaling alone does not close these gaps.
Specialization: High-variance tasks reveal that model choice has an outsized impact on downstream performance. Specialized models (e.g., ENFORMER for regulatory tasks, PLANTCADUCEUS for plant lncRNA) often outperform generalists on their specific domains, even with fewer parameters.

5. Significance and Claims

The paper argues that aggregate leaderboards are unstable and often misleading for genomic model selection. The primary significance of GENEB lies in its ability to:

Expose Trade-offs: Reveal that architectural and pretraining alignment frequently outweigh parameter count, and that no single model dominates across all biological contexts.
Enable Principled Comparison: Provide a shared reference point for assessing progress, moving the field away from fragmented, single-paper evaluations toward controlled, reproducible comparisons.
Guide Model Selection: Motivate a shift toward category-aware model selection. Practitioners are encouraged to select models based on specific task requirements (e.g., regulatory vs. structural tasks) and domain alignment (e.g., eukaryotic vs. prokaryotic) rather than relying on a single global ranking.

The authors conclude that while scaling remains a substantial predictor of aggregate performance, it is insufficient for solving hard genomic tasks or overcoming domain mismatches. Future progress requires complementary advances in pretraining corpus design, inductive biases, and task-specific supervision, guided by the diagnostic insights provided by GENEB.