🧬 biology

On a Regulator-Centric Eligible Universe from Three Curated References, a Perturb-seq Edge Benchmark Has Too Few Evaluable Regulators to Resolve Direct-Edge Recovery from Matched Random

This study demonstrates that current Perturb-seq benchmarks, constrained by a small overlap between curated regulatory references and measured genes, lack sufficient statistical power to distinguish whether causal inference methods successfully recover direct regulatory edges from matched random rankings.

Original authors: Chimdi Walter Ndubuisi

Published 2026-08-03

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Chimdi Walter Ndubuisi

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are a detective trying to solve a massive mystery: how do the thousands of genes inside a living cell talk to each other? Some genes act like bosses (regulators), giving orders to other genes (targets) to turn on or off. If we can map these orders, we get a "Gene Regulatory Network," a blueprint of life's instruction manual. For years, scientists have tried to crack this code using a powerful new tool called Perturb-seq. Think of Perturb-seq as a giant "what-if" machine: scientists use a genetic scissors (CRISPR) to snip out specific genes one by one and then watch, in high definition, how the rest of the cell reacts. It's like pulling a single thread from a sweater to see which other threads unravel.

The big hope was that by feeding this data into computer algorithms, we could automatically reconstruct the exact wiring diagram of the cell. Scientists had already tested these algorithms on tiny, made-up puzzles (simulations) and they worked great. But the real question was: do they work on the messy, complicated reality of a living human cell? This paper is the story of a researcher who decided to put those algorithms to the ultimate test, only to discover that the test itself might be broken. They didn't just check if the algorithms were good; they checked if the game they were playing was even fair.

The Great Map-Making Test That Got Lost in the Crowd

The researcher set out to see if seven different computer programs could correctly identify which genes were giving orders to which other genes. They used data from three different cell types (like different neighborhoods in a city) and compared the computer's guesses against three different "Gold Standard" maps that scientists had already drawn by hand over many years.

But here is where the plot twist happens. The researcher realized they were trying to find a needle in a haystack, but the haystack was so huge that the needle was invisible.

The Haystack Problem
Imagine you are in a stadium with 1,000 people (genes). You want to find out who is friends with whom. The computer algorithms generate a list of millions of possible friendships. However, the "Gold Standard" maps only list a few hundred confirmed friendships. If you ask the computer to guess all millions of pairs, it will look terrible because 99.9% of the guesses will be wrong simply by chance. It's like guessing the winning lottery numbers; even a perfect guesser looks bad if they have to pick from a billion combinations.

To fix this, the researcher decided to only look at the "eligible" friendships: the ones where the "boss" gene was actually one of the genes they had cut with scissors, and the "target" gene was actually in their list. They thought, "Okay, now we have a smaller, fairer test."

The Empty Room Surprise
But when they did the math, they found a shocking problem. Even with this smaller, fairer test, the room was almost empty.

They had data from 20,000 cells.
They had 1,024 genes to look at.
But when they cross-referenced everything with the Gold Standard maps, they found that only 39 unique "boss" genes actually had a confirmed target in their list.

It's like having a library with 20,000 books, but only 39 of them are the ones you are allowed to check out. The rest of the library is off-limits because the maps don't have names for them.

The Result: A Tie with Random Chance
With only 39 bosses to judge, the researcher ran their seven computer programs. They compared the results not just to the Gold Standard, but also to a "Random Guessing" program that just shuffled the names around.

The result? The computer programs could not beat the random guesser.

The average performance of the best programs was +0.0085 points better than random.
But the "confidence interval" (the margin of error) was huge: [-0.0200, +0.0467].
Because this range includes zero, the researcher concluded that the computers were effectively doing the same thing as the random shuffler. They couldn't tell if the programs were smart or just lucky.

One program, the "Vanilla Autoencoder," actually did worse than random, scoring significantly lower. The others were just too close to call. The paper explicitly states that the benchmark was too small to prove the methods worked, but also too small to prove they failed. It was a statistical dead end.

Why the "Perfect" Test Failed

The paper points out that the problem wasn't the computer programs themselves, but the design of the test.

The Maps are Incomplete: The "Gold Standard" maps (TRRUST, DoRothEA, CollecTRI) are like old, hand-drawn maps of a city. They are missing huge chunks of the city because scientists haven't discovered those roads yet. If the computer finds a new road, the map says it's wrong, even if the computer is right.
The Gene List was Wrong: The researcher used a list of 1,024 "highly variable" genes. These are genes that change a lot, but they aren't necessarily the ones the Gold Standard maps care about. It's like trying to find a specific type of fish in a pond, but you only brought a net that catches the wrong kind of fish.
The Sample Size was an Illusion: Even though they had 20,000 cells, the real number of independent tests they could run was only 39. It's like having 20,000 students take a test, but only 39 of them have the answer key. You can't grade the class properly.

The Synthetic vs. Real Reality Check

The researcher also looked at why people thought these programs worked so well in the past. Usually, scientists test them on tiny, made-up puzzles (simulations) with 15 nodes (genes).

The Trap: On these tiny 15-node puzzles, the programs looked amazing. But the researcher found that on these tiny puzzles, even a random guesser would score incredibly high just by luck, because the "positive" answers were so common.
The Fix: When they made the simulation match the real world (same number of genes, same difficulty), the gap between the "perfect" simulation and the "messy" real world shrank, but it didn't disappear. There was still a real, measurable gap where the computers failed to learn the rules as well as they did in the simulation. However, the paper notes that this gap gets smaller as the simulation gets bigger, suggesting that maybe we just need bigger puzzles to see the truth.

The Bottom Line

This paper is a "stop and think" moment for the field of gene mapping.

Did the computers fail? Maybe. But we can't tell yet.
Did the test fail? Yes, definitely. The test was too small and too narrow to give a clear answer.
What should we do? The author suggests that future tests need to be designed differently. Instead of just picking random genes, we need to pick genes that we know are in the Gold Standard maps. We need to build a test where the "eligible" room is full of people, not empty.

The paper concludes that with the current setup, we cannot say whether these methods are magic or nonsense. We simply don't have enough data to decide. It's a reminder that in science, sometimes the most important discovery is realizing that your ruler is too short to measure the object you're holding.

This paper presents a rigorous re-evaluation of the benchmarking framework used to assess causal discovery and gene regulatory network (GRN) inference methods on Perturb-seq data. The central finding is that the current standard benchmark design, when applied to real interventional data, lacks the statistical resolving power to determine whether methods recover direct regulatory edges better than random chance.

Problem Statement

Causal discovery methods have demonstrated success on small synthetic graphs (typically 5–50 nodes), motivating their application to Perturb-seq data, which provides interventional observations via CRISPR perturbations and single-cell RNA-seq. However, evaluating these methods on real data has been problematic. Standard evaluations compare millions of candidate gene pairs against a few hundred curated reference edges. This approach suffers from "prevalence dilution," where the positive rate is so low (<0.01%) that any ranking strategy yields near-zero precision, rendering the metric uninterpretable.

To address this, previous work restricted evaluation to a "regulator-centric eligible universe," considering only edges originating from perturbed genes annotated as regulators in curated databases. While this increased prevalence, this paper argues that the resulting benchmark still suffers from a critical flaw: the number of evaluable regulators is too small to support directional conclusions. The paper supersedes an earlier version of this benchmark that claimed methods outperformed random baselines by 1.4× to 7.4×, a claim the author could not support upon re-analysis.

Methodology

The study benchmarks seven distinct causal discovery and GRN inference methods (plus nine ablations and a random control) across three Replogle et al. [2022] Perturb-seq datasets (K562 genome-scale, K562 essential, and RPE1 essential), each subsampled to 20,000 cells and 1,024 highly variable genes.

Key Methodological Corrections and Designs:

Regulator-Centric Eligible Universe: The evaluation is restricted to edges $(r, t)$ where $r$ is a perturbed gene annotated as a regulator in the reference, $t$ is in the measured gene set, and $t \neq r$ . Crucially, a regulator is only "evaluable" if it has at least one non-self curated target in the measured set.
Counting Corrections: The author corrects previous overestimations of the sample size. While 39–55 regulators were previously cited as candidates, only 0–28 per dataset-reference cell are actually evaluable. The total benchmark rests on 39 distinct regulators across 63 regulator-reference cells.
Statistical Framework:
- Matched Random Baseline: Instead of analytic approximations, the study uses 2,000 matched random permutations per regulator to establish a null distribution.
- Hierarchical Bootstrap: A three-level bootstrap resamples datasets, references within datasets, and distinct regulator identities. This accounts for the fact that the same regulator appears across multiple references (ICC = 0.255), preventing the inflation of effective sample size.
- MixedLM Sensitivity: A linear mixed model with a regulator random intercept is used to assess variance.
Synthetic-to-Real Comparison: To address the mismatch between synthetic benchmarks (often 15 nodes) and real data, the author generates synthetic linear-Gaussian DAGs at five scales (15 to 1,000 nodes). They match the metric (Average Precision), universe rules, and prevalence (via edge density bisection) to the real data to create a fair comparison.
Method Audit: The study identifies implementation mismatches. "GRNBoost2" is excluded from the primary aggregate because its producing code could not be identified. "PC-GES" is revealed to be a partial-correlation proxy, not a true PC or GES algorithm, though a faithful run of PC/GES was attempted on smaller subsets.

Key Results

1. Aggregate Alignment is Indistinguishable from Random
When pooled across the seven distinct methods, the mean Average Precision (AP) above matched random is +0.0085 AP, with a 95% hierarchical confidence interval of [−0.0200, +0.0467].

The interval includes zero, indicating the benchmark cannot distinguish method rankings from random rankings.
A seeded random control arm lands at −0.0013 AP, confirming the pipeline is calibrated.
Only three methods show intervals excluding zero, but all are run on a single dataset (K562-GS), meaning their intervals do not price between-dataset variance. One of these (Vanilla Autoencoder) is significantly below random.
No per-method mean survives Bonferroni correction.

2. The Synthetic-to-Real Gap is Overstated but Real
The previously cited "transfer failure" (methods working on synthetic data but failing on real data) was measured on incompatible axes.

15-Node Benchmark Failure: The standard 15-node synthetic benchmark cannot be prevalence-matched to real data (floor 8.22% vs. real ~2.1%). On this scale, a random ranker scores 4.8× better than the best real method, and NOTEARS scores below chance.
Matched Comparison: When metric, universe, and prevalence are matched at 1,000 nodes, a residual gap remains. The synthetic arm exceeds the real arm by an additive +0.1118 AP (95% CI [0.0746, 0.1489]) on chance-corrected excess AP.
This gap shrinks monotonically with graph scale (from +0.31 at 50 nodes to +0.11 at 1,000 nodes), suggesting the graph size is a material factor, but a genuine transfer failure persists even at large scales.

3. Direct vs. Indirect Discrimination
Methods do score curated direct targets higher than unrelated genes (score ratios 1.22×–21.64×). However, label-ablation experiments (training with shuffled or no perturbation labels) retain similar gradients (1.22×–1.43×). This indicates that a substantial portion of this discrimination is driven by expression correlation rather than perturbation information.

4. Computational Infeasibility of Constraint-Based Methods
Unrestricted PC and GES algorithms timed out (>14,400 seconds) on the 173-gene eligible universe. Adding cells (increasing statistical power) made the problem harder for these algorithms because denser skeletons increase the combinatorial explosion of conditioning sets.

Significance and Claims

The paper explicitly states what it does and does not show:

What it shows: The current benchmark design, constrained by the overlap between curated references and measured gene panels, has an effective sample size of only 39 regulators. At this size, the benchmark cannot resolve whether Perturb-seq methods recover direct edges better than random, nor can it rule it out. The "1.4×–7.4× above random" claim from previous work is withdrawn as it relied on a single regulator in a single cell and misinterpreted the statistical power.
What it does not show: It does not prove that Perturb-seq methods recover no regulatory information; the confidence intervals are wide and include substantial positive effects. It does not claim the methods are equivalent to random, only that they are indistinguishable from random given this specific benchmark's resolving power.
Design Implications: The binding constraint is the number of evaluable regulators, determined by reference-panel overlap, not by cell count. Future benchmarks must report the count of evaluable regulators as the sample size, use explicit matched-random baselines, and select gene panels specifically to maximize overlap with curated references.

In conclusion, the paper reframes the field's understanding of Perturb-seq benchmarking: the failure to demonstrate edge recovery is a finding about the benchmark's resolving power, not necessarily a definitive failure of the methods or the Perturb-seq technology itself. The author calls for a redesign of evaluation instruments to ensure they are capable of answering the questions they pose.