A leakage-aware benchmark for evaluating chemical and cellular generalization in single-cell perturbation-response prediction

Technical Summary: A Leakage-Aware Benchmark for Single-Cell Perturbation-Response Prediction

Problem Statement
Single-cell perturbation models are increasingly utilized for computational screening and experimental prioritization. However, their practical utility is often obscured by ambiguous evaluation standards. Traditional random splitting of data fails to distinguish between distinct predictive tasks: predicting responses to close chemical analogues (local interpolation), transferring knowledge to new cellular contexts, forecasting responses to entirely new chemical scaffolds (structural extrapolation), or handling joint extrapolation across both domains.

The core issue is that perturbation-response records are not independent observations; they form a multilayer relational system where compounds are linked by structure and mechanism, and cellular observations are linked by lineage and context. Random splitting often retains "familiar" neighbors (same drug, same scaffold, or same cell line) in the training set for test records, leading to optimistic performance estimates that do not reflect true generalization capabilities. Existing benchmarks often lack an audit layer to explicitly define what information is available to the model across the train-test boundary.

Methodology
The authors propose a leakage-aware benchmark framework designed to make the evaluation target and the remaining train-test neighborhoods explicit. The framework is implemented using two primary datasets: OpenProblems (a local metadata cohort with 545 treated records) and Sci-Plex 3 (a pseudobulk cohort with 2,200 records).

Key methodological components include:

1. Deterministic Manifests: For each random seed, records are assigned to train, test, or excluded sets based on strict rules.
   • Random Split: Standard 20% test fraction.
   • Scaffold-Held-Out: Groups of records sharing a specific chemical scaffold are held out.
   • Cell-Held-Out: Groups sharing a specific cell line/context are held out.
   • Joint Held-Out: The intersection of selected cell and scaffold groups is held out as the test set. Records satisfying only one condition are excluded to ensure a clean extrapolation task.
2. Record-Level Neighborhood Audits: For every test record, the framework generates audit fields indicating the presence of same-drug, same-scaffold, same-cell, same-dose, and same-Mechanism-of-Action (MoA) neighbors in the training set. It also calculates the maximum Tanimoto similarity to any training drug.
3. Matched-Random Controls: To isolate the effect of the split design from dataset size or composition, the authors generate random controls that match the joint split in terms of test-set size, training-set size, and specific composition variables (e.g., cell context, DE-gene count quartiles).
4. Transparent Probe Models: The study evaluates a suite of "transparent" models to profile information usage, including:
   • Global, cell-context, and drug means.
   • Ridge regression models using cell-context, drug-fingerprints (Morgan vectors), and combinations thereof.
   • A control-conditioned nonlinear benchmark (a pseudobulk adaptation) to test if nonlinearity improves strict extrapolation.
5. Metrics and Uncertainty: Performance is measured primarily by mean row-wise Pearson correlation. Confidence intervals are derived via 10,000 seed-level bootstrap resamples, treating the seed as the inference unit rather than individual records.

Key Results

• Leakage in Random Splits: In OpenProblems, random validation retained same-drug neighbors for 98.9% and same-scaffold neighbors for 97.4% of evaluable test records. Consequently, the cell-context plus drug-fingerprint ridge probe achieved a Pearson correlation of 0.362 under random validation.
• Strict Extrapolation Performance: Under joint cell-plus-scaffold held-out validation, the same probe's performance dropped significantly to 0.118. The paired difference between the random and joint split was 0.218 (95% CI: 0.208–0.228).
• Control Validity: Matched-random controls (matching train/test sizes and composition) yielded scores around 0.336, confirming that the performance drop in joint splits is not due to reduced data size but rather the removal of chemical and cellular neighborhoods.
• Mechanism vs. Structure: Scaffold-held-out evaluation removed structural overlap but retained same-MoA neighbors for 42.7% of annotated test records in OpenProblems, highlighting that structural novelty does not guarantee mechanistic novelty.
• Model Complexity: Simple baselines (e.g., cell-context means) remained stable across random and scaffold splits, whereas fingerprint-based probes declined sharply when chemical neighbors were removed. The nonlinear benchmark showed no improvement in strict extrapolation scenarios compared to linear baselines.
• Sci-Plex 3 Findings: Similar trends were observed in Sci-Plex 3, where the nonlinear benchmark dropped from 0.245 (random) to 0.017 (joint), with a paired difference of 0.228.

Significance and Claims
The paper claims to reframe perturbation-response benchmarking from a simple model-comparison exercise to an evaluation-design problem. The authors assert that:

1. Benchmark scores are conditional estimands: A model's performance is not an absolute property but depends entirely on the specific generalization claim (interpolation vs. extrapolation) and the neighborhood structure permitted across the split.
2. Explicitness is required: A single leaderboard score is insufficient. The framework provides a reusable standard (fixed manifests, audit fields, matched controls) to distinguish between local interpolation, chemical structural extrapolation, cellular extrapolation, and joint extrapolation.
3. Reproducibility: By releasing fixed manifests, source data, and seed-level performance values, the framework allows for reproducible interpretation of whether a score reflects genuine generalization or the exploitation of data leakage.

The study concludes that current benchmarks often conflate different tasks. The proposed leakage-aware framework enables researchers to explicitly define the evaluation target, ensuring that reported generalization claims are supported by the specific relational boundaries of the test set.