🧬 biology

Basal expression predicts context-specific drug responses only where they are reproducible: a leakage-safe 47-line LOCO benchmark on Tahoe-100M

This pre-registered, leakage-safe benchmark demonstrates that the ability of basal transcriptome expression to predict context-specific drug responses is fundamentally limited by the low reproducibility of the target signal, with meaningful prediction only emerging for a small subset of drugs where the residual response is sufficiently reliable.

Original authors: Huazhang Shen

Published 2026-09-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Huazhang Shen

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine a world where doctors could predict exactly how a specific patient's cancer cells would react to a new drug before ever administering it. This is the promise of "virtual cell" models, a rapidly growing field in biology where scientists use computer programs to simulate how living cells behave. The hope is that by feeding a computer the genetic blueprint of a cell line—a group of cells grown in a lab that mimics a patient's tumor—the software could forecast how those cells would change when exposed to a chemical treatment. If successful, this could revolutionize drug development, saving time and money by filtering out treatments that would fail for specific types of cancer. However, a crucial question has lingered in the background: can these models actually learn the unique personality of a cell line, or are they just guessing the average reaction of all cells?

To answer this, an independent researcher named Huazhang Shen conducted a rigorous test using a massive collection of biological data called Tahoe-100M. This dataset contains genetic snapshots of nearly fifty different cancer cell lines, each treated with hundreds of different drugs. The goal was to see if a computer could look at the "basal" state of a cell line—its genetic activity before any drug is added—and accurately predict the unique way that specific line would respond to a drug, distinct from how other lines respond. The researcher set up a strict challenge: the computer had to predict the response of a cell line it had never seen before, using only that line's untreated genetic profile. To ensure the test was fair and free of hidden tricks, the researcher designed a "leakage-safe" benchmark, meaning the computer was never allowed to peek at the treated results of the test line during its learning phase.

The results of this massive experiment were sobering yet illuminating. When the researcher tested six different computer models against the data, none of them could reliably predict the unique response of a cell line for the vast majority of drugs. In fact, for most of the 377 drugs tested, the models performed no better than a simple guess based on the average reaction of all cell lines. The best-performing model managed to find a tiny, barely noticeable signal, but for the overwhelming majority of cases, the computer could not distinguish the unique behavior of a specific cell line from random noise. This failure was not because the computer models were too simple or poorly designed; rather, the problem lay in the data itself.

The researcher discovered that the reason the models failed was that the unique, cell-specific part of the drug response is often not real or consistent. By comparing repeated measurements of the same drug on the same cell lines, the study found that for most drugs, the specific reaction of a cell line is so unstable that it cannot be reliably reproduced. It is as if the cell's response to a drug is a whisper that gets lost in the wind every time you try to listen. If the signal itself is too faint and inconsistent to be heard twice, no amount of advanced computing can predict it. The study established a "ceiling" for prediction: the maximum accuracy any model could ever hope to achieve is limited by how reproducible the biological signal actually is.

However, the story does not end in total failure. The researcher found that for a very small group of drugs—only three out of the hundreds tested—the unique response of the cell lines was strong and consistent enough to be predicted. For these specific drugs, the computer models successfully identified that the cell line's reaction was driven by its fundamental identity. In one case, the model learned that the cell line's sex, determined by the presence of specific Y-chromosome genes, was the key factor in how it reacted to the drug. In another case, the model identified that the cell line's immune and structural programs dictated its response. These findings suggest that while computers cannot predict subtle, fleeting changes in a cell's internal state, they can successfully predict broad, stable characteristics that define a cell line's identity.

Ultimately, this study serves as a vital reality check for the field of virtual cell modeling. It demonstrates that the inability of current models to predict drug responses for most drugs is not a flaw in the algorithms, but a reflection of the biological reality that many drug responses are simply too noisy to predict. The research concludes that for these models to move forward, the scientific community must stop trying to predict the unpredictable. Instead, efforts should focus on the small subset of drugs where the biological signal is clear and reproducible. By acknowledging the limits of what can be predicted, researchers can build more honest and useful tools, focusing their energy on the specific scenarios where the science is actually strong enough to support a prediction.

Technical Summary: Basal Expression Predicts Context-Specific Drug Responses Only Where They Are Reproducible

Problem Statement
The field of perturbation modeling aims to predict how unseen cellular contexts (e.g., specific cancer cell lines) respond to drug treatments. While foundation models and transfer-learning methods increasingly claim success in cross-context generalization, a critical methodological gap remains: few studies rigorously test whether the basal (untreated) transcriptome of a held-out cell line can predict the context-specific component of a drug response.

Current benchmarks often conflate predicting the average drug effect (which is relatively easy) with predicting the residual variation specific to a cell line. The author argues that without isolating the residual (observed response minus the training-context drug mean), models may simply be memorizing drug identities or converging to population means rather than learning true context-specific biology. Furthermore, the reproducibility of these context-specific residuals has rarely been quantified as a "ceiling" for model performance.

Methodology
The author constructed a pre-registered, leakage-safe benchmark using the Tahoe-100M dataset, comprising 47 high-coverage cancer cell lines, 377 drugs (tested at 5 µM), and 12,705 genes.

Experimental Design: A Leave-One-Cell-Line-Out (LOCO) framework was employed. For each of the 47 folds, a single cell line was held out as the test set. The model received only the DMSO (untreated) basal transcriptome of the test line and was tasked with predicting its drug response. Crucially, no treated expression data from the test line was used during training.
Target Definition: The prediction target was defined as the residual ( $R$ ): $R = \text{observed log2FC} - \text{training-context drug-gene mean log2FC}$ . This forces models to predict the deviation from the average drug effect, isolating the context-specific signal.
Models Evaluated: Six models were tested on predicting this residual:
1. no_change: Predicts zero response.
2. drug_mean: Predicts zero residual (the reference baseline, scoring exactly 0 by construction).
3. organ_shrinkage: Predicts the organ-mean residual.
4. knn_transfer: Predicts based on k-nearest neighbors in DMSO PCA space.
5. metadata_ridge: Ridge regression on organ and driver-gene metadata.
6. basal_pc_ridge (Primary): Ridge regression on PCA components of the basal transcriptome.
Metrics: Performance was measured using a reliability-weighted residual $R^2$ . A "validity gate" was implemented where the repeatability ceiling (the correlation of residuals between replicate measurements) was quantified before interpreting model skill.
Statistical Rigor: The study utilized nonparametric bootstrapping, Wilcoxon signed-rank tests against the drug-mean baseline, and permutation nulls (shuffled features) to distinguish true signal from noise.

Key Results

Overall Performance is Negligible: On the full benchmark of 17,561 drug-fold evaluations, no model meaningfully outperformed the drug_mean baseline. The best model (metadata_ridge) achieved a mean weighted residual $R^2$ of only +0.004, while the primary basal_pc_ridge model scored −0.068.
Low Repeatability Ceiling: Analysis of 107 drugs with replicate data revealed that the context-specific residual is largely irreproducible. The median repeat reliability was 0.067, with only 4 drugs exceeding a reliability of 0.2. This low ceiling caps the maximum achievable $R^2$ for any predictor at approximately 0.004 for the typical drug.
Skill Tracks the Ceiling: Model performance strongly correlates with the repeatability ceiling (Spearman $\rho = 0.76$ ). Where the signal is reproducible (reliability > 0.15), models show positive skill.
Three Confirmed Positive Cases: Within the high-reliability bin, only three drugs demonstrated statistically significant predictive skill over the drug-mean baseline:
- Idarubicin ( $R^2 = +0.103$ ): Predictive axis dominated by Y-chromosome genes (sex), distinguishing male from female lines.
- Homoharringtonine ( $R^2 = +0.087$ ): Associated with ECM/inflammatory programs.
- Tucidinostat ( $R^2 = +0.084$ ): Associated with MHC-II/stromal programs.
- Two other drugs in the high-reliability bin (Selinexor, Vinblastine) were indistinguishable from the baseline.
Nature of Predictability: The author concludes that the predictable signal corresponds to cell-line-identity-level phenotypes (e.g., donor sex, immune-stromal programs) rather than subtle regulatory states. The "positive regime" is narrow, relying on just three drugs in this dataset.

Significance and Claims
The paper claims that the failure of context-specific prediction models is primarily a property of the signal's reproducibility, not the inadequacy of the models. The author asserts that:

Benchmarks must report repeatability ceilings: Claims of context-specific prediction on irreproducible targets are unfalsifiable. Benchmarks should report the "ceiling" alongside model skill to contextualize performance.
Current models are not failing; the signal is weak: For most drugs, the context-specific residual is effectively noise. Where a real signal exists (as in the three confirmed drugs), simple linear models on basal expression can recover it.
Predictability is coarse: The predictable features are major phenotypic axes (like sex or stromal composition) rather than fine-grained regulatory states.
Methodological contribution: The study provides a "leakage-safe" 47-line LOCO benchmark and a residualized endpoint design that can be reused for other atlases to rigorously test virtual cell models.

The author explicitly states that their headline answer is "no" for general context-specific prediction, but "yes" for a narrow subset of drugs where the biological signal is sufficiently reproducible. They caution against over-interpreting model skill on targets where the ground truth itself is unstable.

Technical Summary: Basal Expression Predicts Context-Specific Drug Responses Only Where They Are Reproducible

More like this