Technical Summary: AssayBench

Problem Statement

The vision of a "virtual cell"—a computational model capable of predicting cellular responses across diverse perturbations, contexts, and readouts—holds significant promise for accelerating early drug discovery. A critical application of this vision is in silico phenotypic screening, where models predict which genetic or chemical perturbations will modulate a specific phenotype of interest (e.g., cell survival, drug resistance, or infection burden). Unlike tasks focused on reconstructing molecular states (such as single-cell transcriptomics), phenotypic screening operates at the level of assay endpoints, directly aligning with decision-making in drug discovery pipelines.

However, progress in this domain has been hindered by the lack of a standard benchmark. Existing benchmarks for perturbation modeling primarily focus on molecular readouts (e.g., gene expression) and are often trained on specific modalities with structured inputs. They fail to capture the heterogeneity of real-world phenotypic screens, which vary widely in experimental conditions, readout types, and significance criteria. Consequently, there is no established framework to evaluate whether Large Language Models (LLMs) or agentic systems can effectively integrate biological context to predict experimentally measured phenotypic outcomes.

Methodology

Dataset Construction (ASSAYBENCH)

The authors constructed ASSAYBENCH, a benchmark comprising 1,920 publicly available CRISPR screens derived from the BioGRID ORCS database and recent literature. The curation pipeline involved:

1. Harmonization: Normalizing gene symbols to HGNC nomenclature and merging technical replicates.
2. Phenotype Classification: Using LLM-assisted curation to extract detailed phenotype descriptions and effect directions (increase/decrease), grouping screens into five broad categories:
   • Fitness/Proliferation/Viability
   • Chemical/Environmental Response
   • Host-Pathogen/Infection Response
   • Molecular Output/Reporter/Pathway Activity
   • Trafficking/Localization/Structural Phenotypes
3. Scoring: Transforming heterogeneous significance criteria (e.g., p-values, fold-changes) into a continuous relevance score for each gene. Genes satisfying the significance criterion receive positive scores; those associated with the opposite phenotype direction receive negative scores; non-hits receive zero.
4. Temporal Splitting: To induce a realistic distribution shift and prevent data leakage from pretraining corpora, screens were split by publication year:
   • Train: Published before 2021 (1,349 screens).
   • Validation: Published in 2021 (218 screens).
   • Test: Published after 2021 (334 screens).
   • LaTest: A subset of 19 very recent screens (post-September 2025) to probe generalization to unseen literature.

Task Formulation

The task is formulated as a gene-ranking problem. Given a free-text description of the screen (including cell line, library, treatment, and significance criteria), a model must output a ranked list of the top 100 candidate genes. This formulation mirrors real-world workflows where scientists prioritize a short list of candidates for validation, rather than scoring every gene individually.

Evaluation Metrics

To handle the heterogeneity of assays and the varying difficulty of screens, the authors introduced Adjusted normalized Discounted Cumulative Gain (AnDCG@k):

• Condensing: Unassayed genes in the prediction list are ignored rather than penalized as false positives.
• Adjustment: A screen-specific random baseline is subtracted to normalize performance across assays with different inherent difficulties.
• Directionality: The metric accounts for negative relevance scores (genes associated with the opposite phenotype).
• Complementary Metrics: Precision@k and directional False Discovery Rate (dFDR@k) are also reported.

Key Contributions

1. First Large-Scale Phenotypic Benchmark: Introduction of ASSAYBENCH, the first benchmark specifically designed for phenotypic screen prediction, covering 1,920 screens across five phenotype classes.
2. Task Formalization: Casting the problem as screen-specific gene ranking under a temporal split, utilizing a continuous relevance scoring system that harmonizes diverse experimental criteria.
3. Novel Metric: Proposal of AnDCG, a metric tailored to heterogeneous assays that corrects for screen-specific random baselines, enabling fair cross-assay comparison.
4. Comprehensive Empirical Study: An extensive evaluation of frontier generalist LLMs, biology-specific models, trainable neural predictors, and retrieval baselines.

Results

Model Performance

• Frontier LLMs Lead: Zero-shot generalist LLMs (e.g., Gemini 3 Pro, GPT-5.4) outperformed biology-specific LLMs (e.g., C2S-Scale, Biomni), trainable neural gene-relevance predictors, and simple retrieval baselines (Embedding kNN) on the test set.
• Optimization Gains: Techniques such as Supervised Fine-Tuning (SFT), Reinforcement Learning (GRPO), and prompt optimization (GEPA) provided marginal improvements for some models. However, an automatically discovered ensemble of LLM predictions achieved the highest test performance (AnDCG@100 ≈ 0.163), surpassing individual models.
• Performance Ceiling: Despite these advances, all models remain far from the empirical performance ceiling.
  • An "Oracle kNN" (perfect retrieval of the most similar training screen) achieved an AnDCG@100 of ~0.29, roughly 86% higher than the best LLM.
  • A predictor using technical replicates (predicting one replicate based on another) nearly doubled the performance of the best LLM, highlighting that current models have not yet mastered the underlying biological stochasticity.

Generalization and Memorization

• Temporal Shift: Performance dropped on the LaTest split (very recent screens), suggesting that part of the advantage of frontier LLMs on older screens may be driven by memorization of literature present in their pretraining data. Regression analysis confirmed a strong correlation between screen citation count and model performance.
• Scaling Trends: Larger models within the Qwen family generally showed better performance, though gains plateaued, potentially due to the Mixture-of-Experts (MoE) architecture reducing active parameter counts.
• Phenotype Variance: Performance varied significantly by phenotype. Models performed best on "Fitness/Proliferation/Viability" screens (likely due to conserved essential genes) and struggled more with context-specific phenotypes like Host-Pathogen responses.

Biological Biases

Analysis revealed systematic biases in model predictions: GPT models tended to over-represent cell-cycle genes, while Gemini models over-represented developmental biology genes, likely reflecting the distribution of these topics in their training corpora.

Significance and Claims

The paper positions ASSAYBENCH as a practical testbed for measuring progress toward in silico phenotypic screening and, more broadly, virtual cell models. The authors claim that:

• Feasibility: Strong general-purpose LLMs are already competitive for assay-level phenotypic hit prediction without explicit training for the task, suggesting that integrating biological context into language models is a viable path forward.
• Gap to Reality: Current methods are far from solving the task, as evidenced by the large gap between model performance and the empirical ceiling derived from technical replicates.
• Future Directions: The benchmark supports future research in optimizing LLMs for biology (via fine-tuning, ensembling, and prompt engineering) and highlights the need for auxiliary tasks (e.g., perturbation reasoning) and larger datasets to bridge the gap between current capabilities and the requirements of a true virtual cell.
• Benchmark Utility: The authors emphasize the importance of the temporal split and the LaTest set in distinguishing between memorization and genuine generalization, noting plans to update the benchmark quarterly to maintain its utility as models improve.