Dated public evidence and known genes for retrospective evaluation of gene prioritization

In the vast landscape of modern medicine, scientists are constantly trying to predict which genes will become the targets for new drugs. Imagine a researcher standing before a library containing millions of books about human biology, trying to guess which specific volume holds the secret to curing a disease. To make these guesses, they build computer models that learn from past successes: they look at genes that are already known to be effective drug targets and try to find the patterns that made them successful. The hope is that these patterns will help them spot the next great target before a drug is even invented. However, there is a subtle but dangerous trap in this process. The libraries these models use are not static; they are living collections that grow every day. As soon as a gene becomes famous and a drug is developed for it, new information about that gene floods into the databases. If a computer model learns from a database that was updated after a gene became famous, it might simply be memorizing the fact that the gene is famous, rather than learning the biological reasons why it works. This is like trying to predict the winner of a race by looking at a scoreboard that already lists the results; the prediction is technically correct, but it tells you nothing about the runner's actual speed or strategy.

This problem, known as "information leakage," has made it difficult to trust the performance of many gene-prioritization tools. Researchers have long suspected that these tools are overperforming because they are accidentally peeking at the future. To solve this, a team at Pfizer has created a new, meticulously time-stamped dataset that allows scientists to replay history exactly as it happened. Instead of using the current, updated version of public databases, this new resource lets researchers freeze time at any point in the last decade. They can ask the computer model to make a prediction using only the information that was publicly available on a specific date, and then check if the model would have correctly identified the genes that became famous later. This approach strips away the advantage of knowing the outcome in advance, forcing the models to rely on genuine biological signals rather than the noise of accumulated research.

The core of this work is a massive collection of evidence tables covering four major areas of medicine: cancer, inflammatory bowel disease, metabolism, and psoriasis. The researchers gathered data from twenty different public sources, such as clinical trial registries, genetic databases, and protein maps. For every single piece of information in these tables, they attached two critical pieces of metadata: a release date and a "leakage" classification. The release date tells the computer exactly when that piece of information first became public. The leakage classification acts as a filter, sorting the data based on how likely it is to be biased by the history of drug development. Some data points represent pure biology, like the physical structure of a protein or its location in the body, which would exist regardless of whether anyone ever tried to make a drug for it. Other data points represent the history of research itself, such as how many scientific papers have been written about a gene or how many clinical trials have tested it. The latter group is highly suspect because the volume of research often increases only after a gene is already known to be important.

By organizing the data this way, the team created five different "feature sets," or versions of the dataset, ranging from the full collection to a "biology-only" version that excludes all research history and drug-development records. This allows scientists to test their models under different conditions. They can see if a model works well when it is only allowed to see the raw biological facts, or if it only performs well when it is allowed to see the accumulated history of human study. The dataset also includes a list of 19 groups of "known genes," each with a precise date marking when that gene first became a known target or disease association. For cancer targets, this date is particularly strict: it is the day a clinical trial registry first showed that a trial for that specific gene had started, ensuring that the model could not have known about the trial before it was public.

When the researchers tested these tools, they found that the timing of the data matters immensely. They discovered that many features that seem useful today are actually just reflections of how much a gene has been studied. For example, the number of research papers or the number of connections a gene has in a network tends to spike after a gene becomes a drug target. If a model uses these features, it is essentially using the answer key. The new dataset revealed that when you remove these "history" features and look only at the biology that existed before the gene was famous, the performance of many models drops significantly. This suggests that previous evaluations of these tools were likely too optimistic because they did not account for this leakage. The study did not find a single magic bullet that predicts all drug targets, but it did provide a rigorous way to separate genuine biological insight from the noise of research volume.

The researchers also checked the reliability of their own dates and labels. They manually reviewed hundreds of clinical trial records to ensure that the dates they assigned to genes were accurate. They found that while automated systems are generally good at this, they sometimes miss the mark, particularly when a trial is registered after it has already started or when a drug is tested for a different condition than the one being studied. By correcting these dates, they ensured that the "time machine" they built was accurate. They also verified that the biological features, such as the size of a protein's outer surface, remained stable over time and did not suddenly change just because a gene became famous. This stability confirms that the biology-based features are trustworthy and do not suffer from the same time-leakage problems as the research-volume features.

Ultimately, this work provides a new standard for how to test computer models in biology. It shifts the focus from asking "Does this model work on today's data?" to "Could this model have predicted this result if it had only known what was known at the time?" The dataset includes 183 archived releases of public resources, allowing anyone to reconstruct the state of scientific knowledge on any quarter-end from 2016 to 2024. This means that a scientist can now take a new method, run it against the data as it existed in 2018, and see if it would have successfully identified the genes that became targets in 2020. If it fails, the method is not as powerful as it seemed. If it succeeds, the result is much more credible.

The implications of this are profound for the future of drug discovery. By using these dated, leakage-free datasets, researchers can stop building models that simply memorize the history of science and start building models that truly understand the biology of disease. The dataset does not guarantee that a new drug will be found tomorrow, but it ensures that the tools used to find it are not fooling themselves. It offers a way to distinguish between a signal that is truly biological and a signal that is merely a reflection of how much attention a gene has received. In a field where the stakes are high and the data is overwhelming, this clarity is the most valuable tool of all. The researchers have not solved the problem of finding new drugs, but they have built a better compass for the journey, one that points true north rather than toward the places where other travelers have already been.