Publication-date handling policies reshape temporal benchmarks for natural-product target retrieval
This study demonstrates that varying publication-date precision policies in natural product databases significantly alter benchmark construction and query sets, leading to misleading performance gains that stem from expanded relation membership rather than genuine improvements in model retrieval capability.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer
In the world of drug discovery, scientists are constantly trying to predict which natural compounds, like those found in plants, will successfully bind to specific proteins in the human body to fight disease. To test if their prediction tools are actually good, researchers use a method called temporal splitting. Imagine a student taking a history exam: to prove they can learn new facts rather than just memorizing the textbook, they are tested only on events that happened after the textbook was written. Similarly, in drug research, a computer model is trained on data from the past and then tested on new discoveries that were recorded later. This setup is designed to mimic the real-world challenge of predicting future drug candidates. However, the "past" in these databases is not always recorded with perfect precision. Some scientific papers list the exact day a study was published, while others only list the month or the year. This lack of detail creates a gray area: if a study was published in a certain month, does it count as "past" knowledge if the cutoff date falls in the middle of that month?
A team of researchers at the Jiangxi University of Chinese Medicine decided to investigate whether these small differences in how publication dates are handled could drastically change the results of these drug-discovery tests. They focused on a massive database called NPASS, which catalogs the relationships between natural products and their biological targets. The researchers wanted to see if the way they decided which historical records to include in the training data would alter the performance scores of the prediction models. They set up a rigorous experiment where they rebuilt the entire history of known drug-target relationships three different ways. In the strictest version, they only counted records with a specific day of publication. In a more relaxed version, they included any record where the latest possible publication date was before the cutoff, even if the exact day was unknown. They then rebuilt the test scenarios for each version, ensuring that the models were evaluated against the exact same future discoveries.
The results revealed that the choice of date policy fundamentally reshapes the landscape of the test. When the researchers relaxed the rules to include records with only month or year precision, the number of known historical relationships jumped from 4,990 to 13,544. This massive expansion meant that many compounds previously considered "new" or "cold" because they had no historical neighbors suddenly found themselves surrounded by similar past examples. Consequently, the number of difficult test cases, where a model had to guess without any prior similar examples, shrank from 87 down to 48. At first glance, this seemed to suggest that the models performed better under the looser rules, with a noticeable increase in their ability to retrieve the correct targets. However, the researchers dug deeper to understand the source of this improvement.
By carefully comparing the two scenarios, they discovered that the apparent gain in performance was an illusion created by changing the test population, not by the models becoming smarter. When they forced the models to be tested on the exact same set of 48 difficult cases that existed in both the strict and relaxed versions, the performance difference vanished, hovering near zero. The study demonstrated that the improvement came entirely from removing the hardest questions from the test, not from the models learning better. The researchers concluded that without a careful, step-by-step reconstruction of the timeline and a direct comparison of identical test questions, scientists could easily mistake a change in the test difficulty for a genuine breakthrough in drug discovery capability. The study serves as a critical reminder that in the complex machinery of evaluating scientific tools, the rules used to define the past are just as important as the tools used to predict the future.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.