Method Choice, Not Biology, Determines In Silico Perturbation Results: A Systematic Evaluation of Eight Methods Across Four Datasets

Technical Summary: Method Choice, Not Biology, Determines In Silico Perturbation Results

Problem Statement
Current in silico perturbation methods for single-cell transcriptomics suffer from a critical validation gap: they are typically benchmarked on individual datasets, leaving their reliability, generalizability, and ability to recover true causal signals unknown. This lack of systematic cross-method and cross-dataset evaluation makes it difficult to determine whether observed biological conclusions are driven by underlying biology or are artifacts of specific algorithmic choices.

Methodology
The authors conducted a systematic benchmarking study evaluating eight distinct in silico perturbation methods. These methods spanned six different mathematical frameworks and were tested across four diverse datasets. The evaluation strategy included:

• Cross-Method and Cross-Dataset Benchmarking: Comparing performance consistency across different biological contexts.
• Signal Detection Analysis: Assessing the ability of methods to detect transcription factor (TF)-to-pathway directional regulation, specifically focusing on TF-to-glycolysis signals.
• Biological Coherence Checks: Analyzing TF-pathway associations in PBMC monocytes, including enrichment tests for specific pairs (e.g., SPI1 to glycolysis, FOS to AP-1 targets) and using SOX9 as a negative control for pathway specificity.
• Correlation and Ranking Consistency: Measuring the correlation between method rankings to identify anti-correlated results that could reverse biological conclusions.
• Experimental Validation: Validating predictions against CRISPRi Perturb-seq data from K562 cells, specifically comparing predicted perturbation directions against experimental knockdown outcomes for genes such as JUN, CEBPB, SPI1, and FOS.
• Diagnostic Failure Mode Analysis: Utilizing VAE latent space profiling, correlation distribution comparisons, and gene-gene graph analysis to pinpoint specific technical failures in underperforming methods.
• Ablation Studies: Testing whether incorporating a Gene Regulatory Network (GRN) prior into the DDIM method improved target recall.

Key Results

• Widespread Failure of Current Methods: Six of the eight evaluated methods, including widely used Variational Autoencoder (VAE)-based and tensor decomposition approaches, failed to produce detectable TF-to-pathway signals.
• Limited Success: Only CellOracle and DDIM consistently detected TF-to-glycolysis directional regulation. While both methods succeeded in the initial benchmark, CellOracle's predictions failed subsequent experimental validation.
• Method-Dependent Conclusions: The choice of method alone can reverse biological conclusions. For instance, the rankings produced by DDIM and scTenifoldKnk were significantly anti-correlated ($\rho = -0.811$, $p = 0.027$).
• Validation Discrepancy: While CRISPRi Perturb-seq confirmed that TF knockdown suppresses glycolysis gene expression (e.g., JUN $\delta = -1.72$, SPI1 $\delta = -1.57$), CellOracle's predicted perturbation directions showed only 40.9% agreement with experimental directions—a rate not significantly different from chance. This highlights a fundamental gap between steady-state correlation and causal perturbation inference.
• Identified Failure Modes: Diagnostic analyses revealed specific reasons for method failure:
  • VAE Latent Space Competition: Poor signal-to-noise ratios (e.g., STAT3 signal-to-noise of 0.44 vs. SPI1's 4.25).
  • Correlation Noise: TF-glycolysis correlations ($|r|=0.038$) were indistinguishable from background noise ($|r|=0.047$).
  • Graph Non-Specificity: Enrichment factors as low as 0.84x.
• Ablation Findings: Adding a GRN prior to DDIM did not improve target recall ($\delta = 0$ for all TFs), confirming that performance differences are multi-factorial and not solely solvable by prior integration.

Key Contributions

• Systematic Benchmarking: The study provides the first cross-method, cross-dataset evaluation of eight in silico perturbation methods, moving beyond single-dataset validation.
• Empirical Evidence of Method Bias: The paper demonstrates that method selection is a primary determinant of results, capable of generating contradictory biological narratives.
• Diagnostic Framework: It introduces specific diagnostic tools (latent space profiling, correlation distribution comparison, graph analysis) to identify why specific methods fail in particular contexts.
• Validation Gap Identification: The work explicitly quantifies the disconnect between steady-state correlation-based predictions and experimental perturbation outcomes.

Significance and Guidance
The paper establishes that biological conclusions drawn from in silico perturbation are heavily contingent on the chosen algorithm rather than being purely reflective of biological truth. Consequently, the authors provide preliminary, data-driven guidance for method selection to mitigate these risks. This guidance includes:

• Implementing cross-pathway validation.
• Adopting direction-aware benchmarking protocols.
• Adhering to minimum data requirements to ensure robust signal detection, specifically recommending datasets with $\ge$500 cells and $\ge$1,000 Highly Variable Genes (HVGs).

The findings underscore the necessity of rigorous, multi-faceted validation before relying on computational perturbation results for biological inference.