Foundation Models Improve Perturbation Response Prediction

Through an extensive analysis of over 600 models, this paper demonstrates that while foundation models do not universally outperform simple baselines, specific models and integration strategies can significantly improve the prediction of cellular responses to genetic and chemical perturbations when sufficient data is available.

The Gist

Imagine you are a chef trying to predict exactly how a cake will taste if you swap out one ingredient (like changing sugar to honey) or add a new spice. In biology, scientists face a similar challenge: they want to predict how a living cell will react when you "tweak" it—either by turning a gene off (like removing an ingredient) or adding a chemical drug (like adding a spice).

This paper is a massive taste-test competition to see which "recipe books" (AI models) are best at predicting these changes.

Here is the story of what they found, explained simply:

1. The Great "AI Recipe" Debate

For a few years, scientists have been arguing about "Foundation Models." These are huge, super-smart AI systems trained on mountains of biological data (like reading millions of biology textbooks).

• The Skeptics said: "These giant AI models are overkill. A simple calculator or a basic chart works just as well."
• The Believers said: "No way! The AI has learned the secret language of life and can predict things simple tools can't."

The authors of this paper decided to settle the argument by testing over 600 different models to see who actually wins.

2. The Results: Not All AI is Created Equal

They ran the tests on two main types of "ingredients":

• Genetic Tweaks: Changing the cell's DNA (like swapping a gene).
• Chemical Tweaks: Adding drugs or chemicals (like adding a spice).

The Big Discovery:
It turns out, both sides were partially right.

• Some of the fancy AI models were indeed just as good as a simple calculator. They were like a robot chef who forgot how to read the recipe.
• However, other models were amazing. They didn't just guess; they understood the deep connections between ingredients.

The Secret Sauce:
The best-performing models weren't the ones that just read DNA sequences or protein shapes. The winners were models trained on "Interaction Maps" (called Interactomes).

• Analogy: Imagine trying to predict what happens if you remove a player from a soccer team.
  • A model that just looks at the player's stats (DNA) might guess wrong.
  • A model that knows who passes the ball to whom (the interaction map) will know exactly how the team's strategy will collapse.
  • The paper found that models trained on these "social networks" of proteins were the champions.

3. The "Fine-Tuning" Trap

Usually, when you get a powerful AI, you "fine-tune" it (teach it specifically for your task). The researchers tried this, but they found a problem.

• Analogy: Imagine you have a genius chef who knows how to cook 10,000 dishes. You try to teach them to make one specific cake using only 50 samples of that cake.
• The Result: The chef gets confused and starts memorizing the 50 samples instead of learning the general rules. This is called overfitting.
• The Lesson: For biology, sometimes it's better to use the AI's "frozen" knowledge (the general rules it already learned) rather than trying to retrain it on small datasets.

4. The Power of Teamwork (Fusion)

The researchers realized that no single "recipe book" has all the answers. Some know about DNA, some know about protein shapes, and some know about how proteins talk to each other.

• They built a "Team Captain" model (an attention-based fusion model). This model acts like a conductor, listening to all the different AI experts and combining their opinions.
• The Outcome: When they combined the best AI models, the predictions became incredibly accurate. For some cell types, the AI predictions were so good they hit the "theoretical limit"—meaning they were as accurate as the experiments themselves (which always have some noise).

5. The Chemical Challenge

Predicting what happens when you add a drug (chemical) is much harder than changing a gene.

• Analogy: Changing a gene is like swapping a specific Lego brick. Adding a drug is like throwing a handful of mystery dust into the mix; it might stick to one thing, or ten things, or nothing at all.
• The AI models struggled more here. While they could predict genetic changes very well, the "chemical" predictions were still a bit fuzzy. The paper suggests we need better "dictionaries" for drugs that explain how they interact with biology, not just their chemical structure.

The Bottom Line

This paper is a victory for the "Believers," but with a caveat.

• Yes, Foundation Models can revolutionize biology and help us design new drugs faster.
• But, you have to pick the right kind of AI (one that understands biological networks) and know when to use it without over-training it.
• Best Strategy: Don't rely on just one AI. Combine the best ones together, and you get a prediction engine that is nearly perfect.

In short: AI is ready to help us simulate life, but we need to give it the right maps and let the experts work together.

Technical Summary

1. Problem Statement

Predicting cellular responses to genetic (e.g., gene knockouts/overexpression) and chemical (small molecule) perturbations is a fundamental challenge in systems biology and drug discovery. While recent "Foundation Models" (FMs)—large-scale pre-trained models on biological data—have been proposed for this task, the community holds contradictory views. Some studies claim FMs significantly outperform simple baselines, while others argue they offer no improvement over basic statistical methods (e.g., PCA-based linear models).

The paper aims to resolve this debate by conducting a comprehensive, large-scale benchmarking study to determine:

1. Whether FMs genuinely improve perturbation prediction.
2. Which types of embeddings (modalities) are most effective.
3. Whether fine-tuning or integrating multiple FMs yields further gains.
4. How these models perform across different perturbation types (genetic vs. chemical) and datasets.

2. Methodology

The authors conducted an extensive analysis involving over 600 different models across multiple datasets and evaluation metrics.

Datasets

• Genetic Perturbations:
  • Norman: CRISPR activation in K562 cells (large batches).
  • Essential: CRISPR knockdown across 4 cell lines (Hep-G2, Jurkat, hTERT-RPE1, K-562) with ~2,000 genes each. This dataset features small batches, making it a rigorous test for generalization.
• Chemical Perturbations:
  • Sciplex-3: Small molecule treatments on 3 cancer cell lines.
  • Tahoe-100M: A massive dataset (~100M cells) covering 379 drugs across 50 cell lines.

Embedding Strategies

The study evaluated embeddings derived from various modalities:

• Expression-based: Models like scGPT, Geneformer, AIDO.Cell, and scPRINT trained on single-cell RNA-seq data.
• Prior Knowledge/Interactome: Models based on protein-protein interaction networks (STRING), gene ontology, and gene essentiality (e.g., WaveGC, GenotypeVAE, GenePT).
• Sequence/Structure: DNA (AIDO.DNA), Protein sequences (ESM2, ProtT5), and Protein structures (AIDO.StructureTokenizer).
• Chemical: SMILES-based models (ChemBERTa), molecular fingerprints, and target-based embeddings.

Prediction Frameworks

• Task Formulation:
  • Log Fold-Change (LFC) Regression: Predicting the average change in gene expression (Batch-Aware Average Treatment Effect).
  • Differentially Expressed Gene (DEG) Classification: Predicting up/down-regulation status.
• Baselines:
  • Simple Baselines: k-Nearest Neighbors (kNN), Lasso, Linear Regression using PCA embeddings.
  • Advanced Generative Models: Latent Diffusion, Flow Matching, Schrödinger Bridge, and GEARS (Graph Neural Networks).
  • State-of-the-Art: Comparison with STATE, a transformer model designed for cross-context prediction.
• Evaluation Metrics: L2 error for regression, Macro F1 for classification, and a novel "Experimental Error" bound estimated via bootstrapping to define the theoretical performance limit.

Integration and Fine-Tuning

• Fine-Tuning: The authors tested fine-tuning frozen embeddings using "Indexing" (direct mapping) and "In-Silico Knockout" (masking target genes) strategies.
• Multimodal Fusion: An attention-based fusion model was developed to integrate embeddings from diverse sources (e.g., combining sequence, structure, and interactome data) to predict responses.

3. Key Contributions & Results

A. Embedding Modality Matters Most

• Interactome Data is Superior: Contrary to the assumption that massive single-cell expression pre-training is best, embeddings derived from prior knowledge (protein-protein interaction networks, gene function annotations, and textual descriptions) significantly outperformed expression-based FMs.
  • Top Performers: STRING WaveGC, GenotypeVAE, and GenePT (text-based).
  • Underperformers: Pure expression-based FMs (e.g., scGPT, Geneformer) and DNA/Protein sequence models generally performed worse than simple PCA baselines on genetic perturbations.
• Scaling Helps: Larger models within the same category (e.g., AIDO.Cell 100M vs. 3M) showed better performance, suggesting model scaling is beneficial for learning biological representations.

B. Genetic vs. Chemical Perturbations

• Genetic Perturbations: FMs significantly improved predictions. On the K562 cell line, the best embedding closed 77% of the gap between the train mean and the estimated experimental error limit.
• Chemical Perturbations: Results were mixed. In LFC regression, no embedding convincingly outperformed negative controls. However, in the DEG formulation, target-based embeddings (predicting the gene target of a drug and embedding that gene) performed best. SMILES-based models generally underperformed, likely because they are trained on chemical properties rather than biological function.

C. Fine-Tuning vs. Frozen Embeddings

• Risk of Overfitting: Fine-tuning FMs on small perturbation datasets often led to performance degradation due to overfitting.
• Exception: The "In-Silico Knockout" fine-tuning method for AIDO.Cell significantly improved performance, suggesting that task-specific adaptation can work if the architecture aligns with the biological mechanism. However, for most cases, frozen embeddings were more robust.

D. Multimodal Fusion

• Significant Gains: Integrating multiple embedding sources via an attention-based fusion model yielded the best results.
• Approaching Limits: For genetic perturbations in the Essential dataset, the fusion model matched the estimated experimental error for two cell lines (Jurkat, K-562), effectively reaching the theoretical performance ceiling.
• Chemical Limitations: Fusion did not improve performance for chemical perturbations, likely due to the weak signal in individual chemical embeddings and the difficulty of learning complex fusion weights with limited data.

E. Advanced Generative Models

• Complex generative methods (Latent Diffusion, Flow Matching) did not outperform simple kNN regression when using strong embeddings. They were computationally expensive and harder to tune, offering no practical utility for the specific formulation of predicting average treatment effects studied here.

4. Significance and Implications

1. Resolution of the Debate: The paper clarifies that FMs can outperform baselines, but only if the correct modality is chosen. The "foundation" must be biologically relevant (e.g., interactome data) rather than just massive in scale (e.g., raw expression counts).
2. Data Prioritization: The results suggest that for building virtual cell models, interaction data (protein-protein, gene regulatory networks) is more valuable than single-cell expression data alone.
3. Drug Discovery: While genetic perturbation prediction is nearing its theoretical limit with current methods, chemical perturbation prediction remains a challenge. The field lacks a biological-function-aware foundation model for small molecules; current models focus too much on chemical properties.
4. Methodological Guidance: The study advocates for a "embedding-centric" approach where frozen, high-quality embeddings from diverse sources are fused via attention mechanisms, rather than relying on complex generative models or aggressive fine-tuning on small datasets.

Conclusion

The authors demonstrate that foundation models are a powerful tool for perturbation prediction, capable of approaching the limits of experimental noise. However, success depends heavily on the source of the embedding (prior knowledge/interactomes > expression sequences) and the strategy of integration (multimodal fusion). This work provides a roadmap for developing the next generation of "virtual cells" for drug discovery and systems biology.