Constructed negatives determine the population a benchmark can rank over
This paper argues that benchmark rankings are fundamentally limited by the exclusion of poorly annotated entities when constructing negatives, demonstrating that high omission rates (W > 0.5) prevent strict model ordering and urging the community to transparently report target definitions, sampling support, and evidence for comparisons.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer
In the vast landscape of biological discovery, scientists often rely on computer models to predict how proteins interact or where chemical tags attach to them. To teach these models, researchers must show them examples of things that happen (positives) and examples of things that do not happen (negatives). The challenge is that while we have a good list of things that do happen, we rarely have a complete list of everything that does not. Instead, scientists often create a list of "negatives" by assuming that any interaction or modification they haven't seen yet is probably a non-event. This assumption is the foundation of many benchmarks, or standard tests, used to judge how well a new model performs. If the test is flawed, the ranking of the best models becomes unreliable, potentially sending researchers down the wrong path in their search for new drugs or biological insights.
A researcher at Tulane University, Justin Li, has uncovered a fundamental flaw in how these tests are often constructed. He found that the way scientists choose their "negative" examples often leaves out huge portions of the biological world, effectively making the test blind to a majority of the candidates it claims to evaluate. Li's work shows that when a test excludes more than half of the possible candidates, no strict ordering of two models is identified without additional assumptions, even if one model scores higher on the test. The study does not just point out a minor error; it demonstrates that the very population a benchmark claims to rank over is often a tiny, unrepresentative slice of reality, and that improvements seen on these tests can sometimes vanish or even reverse when applied to the full picture.
The core of the problem lies in a rule many researchers use to keep their data clean. To avoid accidentally labeling a real biological event as a "negative" just because it hasn't been discovered yet, scientists often set a high bar for inclusion. They might say, "We will only use proteins as negative examples if we know a lot about them already." While this sounds like a safe way to ensure accuracy, Li showed that this rule acts like a sieve that filters out almost everything. In eight different tasks involving chemical modifications on proteins, this rule excluded between 60% and 97% of the candidate proteins. In the worst cases, the test was only looking at a tiny fraction of the proteins it was supposed to cover.
Li developed a way to measure exactly how much of the target population is missing. He found that once the missing portion exceeds half of the total, the test loses its ability to guarantee a strict ranking between models without assumptions about the excluded region. Imagine trying to judge the quality of a library by only reading the books on the top shelf; if the bottom shelves contain the best books, your judgment will be wrong. In these biological tests, the "top shelf" is the small group of well-annotated proteins, while the "bottom shelves" are the vast majority of proteins that the test ignores. When the ignored portion is this large, a model could appear to be the best on the test simply because it happens to work well on that small, easy group, while failing miserably on the rest of the biological world.
To prove this, Li ran a specific experiment where he compared a standard model against a version that included extra information about the whole protein. On the standard test, which only looked at the well-annotated proteins, the extra information seemed to help. However, when Li reconstructed the test to include the previously excluded proteins, the extra information actually hurt the model's performance in seven out of eight tasks. The feature that looked like a helpful upgrade in the narrow test became a liability when the model was forced to deal with the full range of candidates. This reversal shows that the initial test was not measuring the model's true ability to generalize; it was merely measuring how well the model fit a specific, narrow set of rules.
The study also examined a popular benchmark for protein interactions, which is used to test how well computers can predict which proteins stick together. This benchmark uses a two-step filter to create its negative examples. First, it removes entire proteins that have too many connections, and second, it removes pairs of proteins that are too close in a network structure. Li found that the first step alone excludes entire proteins, meaning that for some proteins, there are no negative examples at all. The second step removes nearly all possible pairs for many other proteins. The result is a test where the "negative" examples are so sparse and biased that they do not represent the real world of protein interactions. In fact, the test excludes 73% of the proteins in its own stated universe, leaving a massive gap between what the test claims to measure and what it actually sees.
Li's work does not suggest that these models are useless, but rather that we must be honest about what they are actually testing. A benchmark should clearly state which part of the biological world it covers and admit what it leaves out. If a test excludes more than half of the candidates, it cannot certify that one model is better than another for the whole population without further assumptions. The researcher recommends that scientists report exactly how many candidates are included and excluded, and they should be cautious about claiming a model is superior unless the test covers a large enough portion of the target to make a fair comparison.
The findings serve as a necessary correction to the field of biological prediction. By separating the performance of a model on the data it sees from its performance on the data it misses, Li provides a clearer path forward. The study suggests that the current practice of using strict filters to create "clean" negatives may be doing more harm than good, creating an illusion of progress that disappears when the models face the messy, incomplete reality of biological data. The solution is not to abandon these tests, but to design them with a full understanding of their limits, ensuring that the rankings they produce reflect the true potential of the models rather than the quirks of the test itself.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.