🧬 biology

An Empirical Comparison of Virtual Cell Models: Perturbation Prediction, Representation, and the Baseline Gap

This paper presents a unified benchmark of eleven virtual cell models, revealing that while deep learning approaches significantly outperform baselines in representation tasks and combinatorial perturbation prediction, they generally fail to surpass simple linear models in predicting unseen single-gene perturbations, suggesting the field has achieved a "virtual microscope" for cell state analysis rather than a true "virtual simulator" for causal perturbation dynamics.

Original authors: Olivia Denvis

Published 2026-07-22

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Olivia Denvis

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are trying to build a "digital twin" of a living cell. In the real world, scientists can now take pictures of millions of individual cells and see how they react when you poke them with a genetic switch or a chemical drug. The big dream in this field is to create an AI Virtual Cell: a super-smart computer program that doesn't just memorize these pictures, but actually understands how a cell works. If you tell this AI, "Hey, turn off this specific gene," it should be able to predict exactly how the cell will change, even if it has never seen that specific gene turned off before. It's like having a crystal ball for biology, one that could help us design new medicines or understand diseases without needing to test every single possibility in a lab.

To understand if these AI crystal balls are real or just magic tricks, we need to know a few things about how they are tested. First, there are perturbations, which are just fancy words for "changes" or "interventions" you make to a cell, like deleting a gene or adding a drug. Second, there are baselines, which are the "control group" of the experiment. In this case, the simplest baseline is just guessing that the cell won't change at all, or guessing that the change will be a simple sum of parts. Finally, there are metrics, which are the scoring systems used to grade the AI. If you grade a student only on how well they can recite a textbook they've already read, they might get an A, but that doesn't mean they can solve a new math problem. The question scientists are asking is: Do these fancy AI models actually solve the new problems, or are they just really good at reciting the textbook?

The Great AI Cell Showdown: Microscopes vs. Simulators

A researcher named Olivia Denvis decided to put the most popular "Virtual Cell" models to the ultimate test. Instead of letting each model use its own rules, data, and scoring system (which makes comparing them like comparing apples to oranges), she built a giant, fair arena. She gathered eleven different AI models—ranging from simple math equations to massive, complex neural networks—and threw them all into the same six different challenges using nine different datasets. It was a massive "battle royale" to see which AI could actually predict how a cell reacts to a poke, and which ones were just bluffing.

Here is the twist: The results were a bit of a shocker.

The "Crystal Ball" is actually a "Microscope"
The biggest finding is that for the most common task—predicting what happens when you mess with a single gene—these fancy, expensive AI models are barely better than a simple, cheap math trick. The paper found that a "well-tuned linear baseline" (think of it as a simple calculator that just adds up the effects) performed almost as well as the most advanced AI, which was trained on hundreds of millions of cells.

In fact, the best AI model in the bunch, called State, only improved on the simple calculator by 26%, while the simple calculator itself was already 19% better than just guessing "nothing happens." Some of the other massive AI models, which cost thousands of hours of computer time to train, actually performed worse than the simple calculator. It's like hiring a team of rocket scientists to do your taxes, only to find out a calculator app does the job just as well, and the rocket scientists are actually making mistakes.

The Trap of the "All-Gene" Score
One of the most important lessons from this paper is about how we grade these models. Many previous studies used a scoring system called "all-gene Pearson correlation." The paper explains that this score is a bit of a trap. Because most genes in a cell don't change when you poke it, an AI that just predicts "nothing changes" gets a huge score on this metric. It's like a student who refuses to answer any questions on a test but gets an A because the teacher only graded them on the questions they didn't have to answer.

The paper shows that when you switch to better scores that focus on the genes that actually change (like the "top-20 differentially expressed genes"), the "do-nothing" AI drops to the bottom of the class, and the real differences between the models become clear.

Where the AI Actually Shines
So, are these models useless? Not at all! They just aren't the "simulators" everyone hoped they were yet. They are excellent "microscopes."

Representation (The Microscope): When the task is simply to describe or categorize a cell (like saying "this is a liver cell" or "this is a cancer cell"), the massive AI models are incredible. They beat the simple calculators by a wide margin. They are great at organizing the messy data of life into neat, understandable groups.
The Hard Stuff (The Simulator): The AI models only start to beat the simple calculators in very specific, difficult situations:
- Combinatorial Perturbations: When you mess with two genes at once, and the result isn't just the sum of the two (a "non-additive" effect), the AI's ability to learn complex patterns helps.
- Chemical Doses: When predicting how a cell reacts to different amounts of a drug, the AI handles the complexity better than a simple line.
- Cross-Context Transfer: When you try to predict how a drug works in a cell type the AI has never seen before, the AI's "pre-trained" knowledge helps it adapt better than a simple calculator, though it still struggles.

The Bottom Line
The paper concludes that the field is currently closer to having a Virtual Microscope than a Virtual Simulator. We have amazing tools to look at and organize cells, but we don't yet have a tool that can reliably simulate how a cell will react to a new intervention better than a simple math model.

The authors also point out the cost of this "magic." The most advanced model, State, required about 42,000 GPU-hours to train. That is a massive amount of computing power. The simple linear baseline, which performed nearly as well for single-gene predictions, took less than 0.1 GPU-hours. The paper suggests that until we can figure out how to make these models actually simulate causal logic (understanding why a gene does something, not just what it usually does with other genes), the expensive AI models might be overkill for simple prediction tasks.

In short, the "AI Virtual Cell" is a powerful tool for understanding what a cell is, but it's not quite ready to be a crystal ball for predicting exactly what a cell will do next. The race is on to build a simulator that can truly outsmart the simple calculator, but for now, the calculator is holding its own.

Technical Summary: An Empirical Comparison of Virtual Cell Models

Problem Statement
The field of computational biology has coalesced around the vision of an "AI virtual cell" (AIVC): a learned model capable of representing cellular states and simulating responses to genetic and chemical perturbations. While a diverse "zoo" of candidate models exists—ranging from purpose-built perturbation predictors (e.g., GEARS, CPA, scGen) to foundation-model embeddings (e.g., Geneformer, scGPT, scFoundation) and integrated systems (e.g., State)—their capabilities remain unclear due to heterogeneous evaluation practices. Models are trained on different datasets, use varying data splits, and report disparate metrics, making cross-paper comparisons impossible. Furthermore, there is a lack of consensus on whether complex deep learning architectures genuinely outperform simple, interpretable baselines, particularly given recent critiques suggesting that deep models may not surpass linear predictors on unseen single-gene perturbations.

Methodology
To address these inconsistencies, the authors constructed a unified benchmark evaluating eleven methods across five model families. The study holds data splits, preprocessing pipelines, and evaluation metrics constant to ensure a fair comparison.

Models Evaluated:
- Simple Baselines: Cell-mean (predicting the unperturbed control profile) and Linear/additive (ridge-regularized linear mapping of perturbation identity to expression shifts).
- Generative Models: scGen, CPA, chemCPA (latent variable models).
- Graph Models: GEARS (utilizing gene-gene structure).
- Foundation Embeddings: Geneformer, scGPT, scFoundation, UCE (pretrained on atlas-scale corpora, adapted via fine-tuning or in-silico intervention).
- Integrated Model: State (coupling State Embedding and State Transition transformers trained on ~170M observational and ~100M perturbational cells).
Datasets and Tasks: The benchmark spans nine public datasets covering genetic Perturb-seq (Adamson, Norman, Replogle), chemical screens (sci-Plex, Tahoe-100M), and representation tasks (cell-type annotation, batch integration).
Evaluation Splits: Three generalization scenarios were tested:
1. Unseen Perturbation: Entire genes or drugs held out.
2. Combinatorial: Single genes seen, but specific gene pairs held out to test non-additive interactions.
3. Cross-Context: Entire cell types or lines held out to test transferability.
Metrics:
- Perturbation Prediction: Delta-Pearson on top-20 differentially expressed (DE) genes ( $p\Delta$ ), Differential-Expression Score (DES), Perturbation-Discrimination Score (PDS), Mean Absolute Error (MAE), and Energy Distance. The study explicitly critiques the widespread use of "All-gene Pearson correlation," demonstrating its susceptibility to trivial baselines.
- Representation: Cell-type annotation accuracy and scIB integration scores.

Key Results

Perturbation Prediction vs. Linear Baselines:
On unseen single-gene perturbations, no deep learning method dramatically outperforms a well-tuned linear/additive baseline. The best-performing model, State, improved over a cell-mean baseline by 26%, compared to 19% for the linear baseline and 22% for GEARS. Several foundation-model predictors (e.g., scGPT) fell below the linear baseline. The margin between the best deep model and the linear baseline is small (0.043 in $p\Delta$ ) and often within the noise of cross-split variation.
Metric Sensitivity and Pitfalls:
The study identifies a critical evaluation flaw: All-gene Pearson correlation is maximized by models that predict "no change" (the cell-mean baseline), as most genes do not change under perturbation. This metric fails to distinguish useful predictors from useless ones. In contrast, metrics focusing on changes ( $p\Delta$ , DES, PDS) provide sensible rankings where the linear baseline is competitive but not dominant.
Where Deep Models Excel:
Deep models demonstrate clear, reproducible gains in specific niches:
- Combinatorial Perturbations: On non-additive genetic interactions (synergy, suppression, neomorphism), deep models (State, GEARS) significantly outperform additive baselines, which fail to capture these complex interactions.
- Chemical Perturbations: Models like State and chemCPA show wider margins over linear baselines on dose-response data (sci-Plex), though performance drops sharply in cross-cell-line settings.
- Cross-Context Transfer: While all models suffer from distribution shift when predicting responses in unseen cell types, models with strong pretrained priors (State, scGPT) retain a higher fraction of their performance (67–69%) compared to linear baselines (49%).
Representation vs. Prediction:
A stark dichotomy exists between representation and prediction capabilities. Foundation embeddings decisively beat linear baselines on representation tasks (cell-type annotation and batch integration), achieving 6–8 percentage points higher accuracy on difficult datasets. Conversely, these same models struggle to beat linear baselines on perturbation prediction.
Compute Efficiency:
The linear baseline achieves ~73% of the best deep model's improvement over the cell-mean baseline at a negligible training cost (<0.1 GPU-hours vs. ~42,000 GPU-hours for State). The "accuracy-compute frontier" for perturbation prediction is dominated by the cheap linear baseline, whereas representation tasks justify the high cost of foundation models.

Significance and Claims
The paper argues that the field is currently closer to a "useful virtual microscope" than a "virtual simulator." While foundation models have successfully solved the representation problem (encoding organized, largely correlational cell-type structures), they have not yet mastered the simulation of counterfactual perturbations, which requires causal regulatory logic rather than just correlation.

The authors claim three primary contributions:

Standardization: Providing a unified benchmark that reveals the true relative performance of diverse model families under fixed conditions.
Evaluation Correction: Demonstrating that the field's reliance on all-gene correlation masks the failure of models to predict actual biological changes, and establishing that simple linear baselines are the true bar for perturbation prediction.
Realistic Expectations: Clarifying that deep models offer genuine value primarily for non-additive interactions, cross-context transfer, and representation, rather than for general single-gene perturbation prediction.

The study concludes that the "virtual cell" vision remains split: representation is a mature regime where pretrained models excel, while simulation remains an open challenge where simple, interpretable baselines remain highly competitive. The authors release the benchmark code and splits to ensure future models are measured against this "unforgiving bar."

The Great AI Cell Showdown: Microscopes vs. Simulators

Technical Summary: An Empirical Comparison of Virtual Cell Models

More like this