🧬 biology

From ceilings to corrected estimates: capture–recapture on reference networks rescales, but rarely reorders, gene regulatory network benchmarks

This study demonstrates that while reference database incompleteness severely limits the absolute accuracy of gene regulatory network benchmarks, applying capture–recapture-based corrections reveals that the relative ranking of inference methods remains robust and largely unchanged despite significant uncertainty in the estimated size of the true interactome.

Original authors: Liu Chen

Published 2026-08-11

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Liu Chen

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine trying to grade a student's essay, but the teacher's answer key is missing half the correct answers. If the student writes a brilliant sentence that isn't on the key, the teacher marks it wrong. The student's grade drops, not because they wrote poorly, but because the key was incomplete. This is the daily struggle of scientists trying to map the "wiring diagram" of life, known as a Gene Regulatory Network (GRN). These networks are like the instruction manuals for cells, showing which genes turn other genes on or off. To see if a computer program can figure out these connections, scientists compare its guesses against a "gold standard" list of known connections. But here's the catch: that gold standard list is a work in progress, missing thousands of real connections. For years, scientists have known their scores were too low because of this missing data, but they didn't know how low, or if the missing data was unfairly hurting some computer programs more than others. It's like knowing your test score is wrong, but not knowing if you're actually the top student or just the one who got unlucky with the missing questions.

This paper, titled "From ceilings to corrected estimates," tackles that exact mystery. The author, Liu Chen, uses a clever statistical trick called "capture–recapture"—a method originally designed to count fish in a lake or animals in a forest—to estimate just how incomplete these gene lists really are. By treating three different gene databases as three separate "nets" cast into the same ocean of biological truth, the study asks: how many fish are we missing? The results are a mix of surprising uncertainty and reassuring stability. The study finds that the total number of real gene connections is likely much higher than we thought, with our current lists capturing only about 5% of the truth (ranging roughly between 3% and 10%). However, the most important discovery is that while the scores of the computer programs were way too low, the ranking of who is the best program didn't change at all. Even after fixing the math to account for the missing data, the same programs stayed at the top and the same ones stayed at the bottom. The paper proves that the "missing list" problem makes the numbers look bad, but it doesn't trick us into thinking the wrong programs are the best.

The Setup: The Missing Map

To understand the story, you need to know a few things about how scientists study genes. Inside every cell, genes don't work alone; they talk to each other. Some genes act as managers, turning other genes on or off. Scientists call this a "gene regulatory network." To build a map of these connections, they use computer programs that look at data from cells and guess which genes are talking to which.

But how do you know if the computer is right? You need a reference map, a "ground truth," that lists all the connections we already know for sure. Scientists have spent decades building these lists by reading thousands of research papers and running experiments. They have three main lists they use for testing: one called STRING, one called BioGRID, and one called TRRUST.

The problem is that these lists are like a library that is missing most of its books. We know there are millions of connections in a human cell, but these lists only have a few thousand. When a computer program guesses a connection that is real but not on the list, the list says, "That's wrong!" This makes the computer look bad, even if it's actually doing a great job. Scientists call this the "ceiling problem": because the list is incomplete, no computer can ever get a perfect score, no matter how smart it is.

The Detective Work: Counting the Missing Fish

The author of this paper decided to stop guessing and start counting. They used a method called capture–recapture. Imagine you are trying to count how many fish are in a pond.

You cast a net (List A) and catch 100 fish. You tag them and put them back.
You cast a second net (List B) and catch 100 fish. 20 of them have tags from the first net.
You cast a third net (List C) and catch 80 fish. Some have tags from A, some from B, and a few have tags from both.

By looking at how many fish appear in one net, two nets, or all three, you can estimate how many fish were in the pond that none of the nets caught.

In this study, the "fish" are gene connections, and the "nets" are the three databases (STRING, BioGRID, and TRRUST). The author cast these nets over a specific set of 359,700 possible gene pairs.

The Catch: The three lists together only found 2,511 connections.
The Overlap: The lists overlapped a lot more than chance would predict. For example, STRING and BioGRID shared 139 times more connections than you would expect if they were independent. This is because they often pull data from the same popular research papers.

The Big Numbers: How Incomplete Are We?

The author ran the math to estimate the total number of real connections (the "latent interactome"). The results were a bit shaky because the lists are so dependent on each other.

The Range: Depending on how you model the overlap, the total number of real connections could be anywhere from 2,579 to 46,864.
The Percentage: This means our current lists only capture between 5.4% and 54.4% of the truth. The author's best guess is that we are only seeing about 5% of the real connections.

This confirms that the "ceiling" is very low. If a computer program gets a score of 0.013 (which looks terrible), it might actually be performing at a level that would be 0.245 if we had a perfect list. The scores are being crushed by the missing data.

The Twist: Does the Ranking Change?

Here is the most exciting part. Scientists were worried that the missing data might be unfair. Maybe some computer programs are better at guessing the connections that are missing from the lists, while others only guess the easy ones that are already on the lists. If that were true, the missing data would be lying to us, making a bad program look good and a good program look bad.

The author tested this by applying a "correction" to the scores. They used the capture–recapture data to figure out that the lists are more complete for famous, well-studied genes and less complete for obscure ones. They then adjusted the scores to account for this bias.

The Result: The ranking of the eight computer programs did not change.

The program that was #1 before the correction stayed #1.
The program that was #8 stayed #8.
The order remained exactly the same, even though the scores themselves jumped up by about 14 to 20 times.

The author proved mathematically that if you just multiply everyone's score by the same number (like dividing by the 5% completeness), the ranking stays the same. The only way the ranking would change is if the programs were guessing different types of missing connections. But in this case, the programs were all guessing the same types of connections, just with different levels of accuracy.

The Simulation: Proving the Tool Works

To make sure their math wasn't broken, the author ran a simulation where they knew the real answer. They created fake data where the lists were biased against certain types of connections.

In this fake world, the uncorrected scores got the ranking wrong.
When they applied their correction, the ranking fixed itself and matched the truth.

This proved that their method can fix a broken ranking if the bias is strong enough. The fact that it didn't change the real-world ranking means the real-world bias wasn't strong enough to trick the ranking in the first place.

The Takeaway

This paper tells us two main things:

The numbers are meaningless: When you see a gene network score like "0.013," don't panic. It's just a number based on a broken ruler. The real performance is likely 15 to 20 times better.
The rankings are safe: Even though the ruler is broken, it's broken in the same way for everyone. The computer program that is currently the best is still the best, and the one that is worst is still the worst. The missing data makes us look bad, but it doesn't make us wrong about who is winning.

The study also found that the choice of which list you use matters more than how incomplete it is. If you test a program against a list of physical connections (STRING), it might win. If you test it against a list of regulatory connections (TRRUST), it might lose. This suggests that scientists need to be careful about which "gold standard" they use, because the list itself changes the winner more than the missing data does.

In short, the map of life is still very incomplete, but our compass for finding the best tools to read that map is working just fine.

Technical Summary: From Ceilings to Corrected Estimates

Problem Statement
Gene regulatory network (GRN) inference benchmarks rely on comparing predicted edges against curated reference databases (e.g., STRING, BioGRID, TRRUST). These databases are known to be radically incomplete, meaning the "ground truth" is a partial subset of the actual interactome. Recent work established that this incompleteness imposes a computable upper bound (ceiling) on performance metrics like the Area Under the Precision-Recall Curve (AUPRC). However, a ceiling only indicates that scores are too low; it does not quantify the magnitude of the error, nor does it determine whether the incompleteness distorts the ranking of different inference methods. If curation efforts are biased toward well-studied genes, methods that concentrate predictions on these genes may be penalized differently than those with broader prediction distributions, potentially rendering benchmark rankings artifacts of the reference data rather than true method performance.

Methodology
The author proposes a three-step framework to move from a theoretical bound to a de-biased point estimate:

Capture–Recapture Estimation of Completeness:
Treating three major reference databases (STRING-physical, BioGRID multi-validated, and TRRUST) as partially overlapping "captures" of a single latent interactome, the author estimates the completeness ( $\pi$ ) of the union of these lists.
- Modeling Dependence: Recognizing that these databases are not independent (they share curation practices, literature sources, and import data from one another), the author rejects the Lincoln–Petersen estimator. Instead, they fit a family of log-linear models with explicit two-way interaction terms to account for positive dependence.
- Stratification: To address the "Selected at Random" (SAR) assumption, where labeling probability depends on covariates, the author stratifies the candidate gene pairs by "literature attention" (PubMed count). They fit pooled-dependence models within these strata to estimate stratum-specific completeness profiles.
Positive–Unlabelled (PU) Correction:
Using the estimated completeness, the author applies a correction to the observed AUPRC.
- Under the "Selected Completely at Random" (SCAR) assumption, the correction is a simple division by the global completeness $\pi$ .
- Under the weaker SAR assumption, they employ a propensity-weighted correction (Horvitz–Thompson style), weighting observed edges by the inverse of their stratum-specific completeness.
Theoretical Analysis of Ranking Stability:
The author proves two key propositions regarding the effect of these corrections on method rankings:
- Proposition 1: Any constant-propensity correction (global $\pi$ ) rescales the AUPRC of all methods by the same factor and therefore cannot change their relative ordering.
- Proposition 2: A propensity-weighted correction is invariant to a common rescaling of the propensity function. Thus, the ranking depends only on the shape of the completeness profile across strata, not its absolute level.

Key Results

Weak Identification of Absolute Completeness: The absolute size of the latent interactome is poorly identified. Depending on the assumed dependence structure, the estimated number of true edges ranges from ~2,500 to ~47,000 (implying completeness $\pi$ between 5.4% and 54.4%). Even with a fourth diagnostic list (HuRI), the estimate remains uncertain by a factor of roughly two to three.
Robust Identification of Relative Profiles: Despite the uncertainty in the absolute level, the shape of the completeness profile across literature-attention strata is highly stable. Three distinct dependence models, which disagree on the absolute level by a factor of three, agree on the relative completeness profile to within a few percentage points.
Invariance of Benchmark Rankings:
- Applying the propensity-weighted correction rescales the absolute AUPRC values upward by an order of magnitude (e.g., from ~0.01 to ~0.20), making the numbers interpretable as estimates of true performance.
- Crucially, the correction does not reorder the eight evaluated GRN inference methods (including scGPT variants, GENIE3, GRNBoost2, and co-expression baselines) in the primary PBMC dataset. The ranking remains identical under four different propensity models.
- In the replication MALT dataset, only one adjacent, statistically indistinguishable pair (GENIE3 and scGPT-emb) swapped ranks, a change not considered statistically meaningful.
Synthetic Validation: In a controlled simulation where the ground truth is known, the propensity correction successfully repaired mis-orderings caused by strong covariate-dependent labeling bias. The failure to reorder methods in real data is attributed to the fact that the real-world labeling bias (a ~3x gradient in completeness) is not steep enough relative to the similarity of current GRN methods to cause a rank inversion.
Reference Choice vs. Completeness: The choice of reference database (STRING vs. BioGRID vs. TRRUST) had a far greater impact on the resulting rankings than the correction for incompleteness. For instance, scGPT's gene embeddings performed best against STRING but poorly against TRRUST, reflecting the difference between physical interaction and transcriptional regulation.

Significance and Claims
The paper concludes that while the absolute performance numbers in GRN benchmarks are currently uninterpretable due to severe reference incompleteness, the relative ordering of methods is robust. The published rankings are unlikely to be artifacts of partial labels because:

Global corrections (which are the only ones that could theoretically reorder methods if the level were known) mathematically cannot change rankings.
Propensity-weighted corrections, which could change rankings, depend on the relative profile shape, which is well-identified and stable.
Current GRN methods, which largely rely on similar covariance structures in expression data, do not differ sufficiently in their distribution of predictions across strata to trigger a rank inversion under realistic bias conditions.

The author argues that the primary threat to benchmark comparability is not the degree of incompleteness, but the choice of reference database. They suggest that benchmarks should report scores against individual references rather than pooled "ground truth" to avoid masking fundamental differences in what different methods are actually predicting.