Model Rankings in Drug–Target Affinity Prediction Vary Across Evaluation Regimes: A Controlled Multi-Seed Study of Model-Comparison Reliability

In the race to discover new medicines, scientists often face a daunting task: finding the right key for a specific lock. The "keys" are small molecules, potential drugs, and the "locks" are proteins inside the human body that drive disease. When a drug fits well into a protein, it can stop a disease process or fix a broken function. Predicting how tightly a drug will bind to a protein is a critical step in screening millions of candidates, allowing researchers to focus their resources on the most promising ones. For years, computer models have been trained to make these predictions, learning from vast databases of known drug-protein interactions. The goal has always been to build a model that can accurately guess the strength of a bond it has never seen before, whether that means predicting how a new drug will behave against a known protein, or how a known drug might interact with a brand-new protein target.

However, a new study from Harbin Normal University suggests that the answer to "which model is best" depends entirely on the question being asked. The researchers, Haobo Kui and Xinyuan Cui, set out to test whether the rankings of these computer models remain stable when the rules of the game change. In the real world of drug discovery, a model might be asked to predict interactions for a drug that has never been seen before, or for a protein that has never been seen before, or for a mix of both. The team discovered that a model that wins in one scenario can lose in another, and that the order in which models are ranked can flip completely depending on the specific testing conditions. Their work, which involved running over a hundred carefully controlled experiments, reveals that there is no single "best" model for all situations; instead, the right choice depends on matching the model to the specific generalization challenge it will face.

To understand the scale of this investigation, imagine a laboratory where scientists are testing five different types of engines. In a standard test, they might drive all engines on a smooth, familiar track. But in the real world, some engines need to handle icy roads, while others must navigate steep mountain passes. Kui and Cui built a rigorous testing framework to see if the engine that wins on the smooth track also wins on the ice or the mountains. They used two massive, cleaned-up databases of drug-protein interactions, containing over 118,000 records from one source and over 30,000 from another. From this data, they created three distinct testing environments. The first was a "warm" setting, where the model was tested on drugs and proteins it had already seen during training, essentially measuring how well it could remember and interpolate what it had learned. The second was a "cold-drug" setting, where the model had to predict interactions for entirely new chemical compounds it had never encountered. The third was a "cold-target" setting, where the model faced entirely new protein sequences, simulating the challenge of finding a drug for a newly discovered disease mechanism.

The researchers did not simply run the models once and take the results. To ensure their findings were not just a fluke of random chance, they ran each experiment three times with slightly different starting conditions, a technique known as using multiple "seeds." They compared three different ways of combining drug and protein information, along with two established baseline methods, ensuring that all models had roughly the same amount of computing power. In total, they conducted 126 separate training runs, keeping the data splits and evaluation rules identical across the board. Crucially, they kept the final test results sealed until every single run was complete, preventing any accidental bias where the model might be tweaked to fit the test data. This strict, auditable approach allowed them to trace every result back to its source, ensuring the integrity of their comparison.

The results were striking and counterintuitive. When the researchers looked at the rankings of the models under the "warm" conditions, one model consistently outperformed the others. However, when they switched to the "cold-drug" scenario, the rankings shifted dramatically. In the most extreme case, involving the larger dataset, the order of the models for the "cold-target" scenario was the exact opposite of the order for the "cold-drug" scenario. A model that was the clear winner for predicting interactions with new proteins became the clear loser when asked to predict interactions with new drugs. This complete reversal, measured by a statistical tool that compares rankings, showed that a model's success is not an absolute quality but a relative one that changes with the context.

Furthermore, the study highlighted that even within the same testing scenario, the results were not always perfectly stable. When looking at the individual runs rather than just the average, the researchers found that the exact order of the models could vary from one training run to the next. This means that relying on a single experiment to declare a winner is risky, as the outcome could be influenced by the random noise of the training process. By running multiple seeds, the team could see the full picture of how reliable a model's performance truly was. They found that while some models showed consistent advantages in specific areas, such as the "cold-drug" setting where certain fusion strategies performed better across both datasets, no single model dominated every metric and every scenario.

The researchers also explored how these findings translate to a real-world application by looking at a specific, unseen protein target called VEGFR2, which is involved in blood vessel growth and is a major target for cancer therapies. Using the frozen predictions from their experiments, they simulated how different models would rank potential drug candidates for this specific target. They found that the models produced different lists of top candidates. One model might rank a specific compound as the best choice, while another might place it much lower. This demonstrated that the choice of model is not just an academic exercise; it directly impacts which drug candidates scientists would choose to test in a lab. If a researcher picks a model based on its performance in a "warm" setting, they might end up prioritizing the wrong compounds for a new target.

Ultimately, this study serves as a vital guide for the future of computational drug discovery. It argues against the idea of a universal champion model that can be applied to every problem. Instead, it suggests that the reliability of model comparisons depends on matching the evaluation method to the intended use. If the goal is to find drugs for new targets, the model must be tested on new targets. If the goal is to find new drugs for known targets, the model must be tested on new drugs. The researchers conclude that for model selection to be truly reliable, it must be scenario-matched, multi-seed, and auditable. By understanding that rankings shift across different evaluation regimes, scientists can make more informed decisions, ensuring that the tools they use are fit for the specific challenges of the drug discovery pipeline.