Benchmarking Single-Pose Docking, Consensus Rescoring, and Supervised ML on the LIT-PCBA Library: A Critical Evaluation of DiffDock, AutoDock-GPU, GNINA, and DiffDock-NMDN

This study evaluates various docking and scoring methods on the LIT-PCBA library, finding that while supervised machine learning re-ranking significantly improves enrichment over single methods like AutoDock-GNINA, no single approach dominates across all targets, highlighting the continued modest performance of current virtual screening workflows on realistic datasets.

The Gist

Imagine you are a treasure hunter trying to find 10,000 specific, rare gems hidden inside a massive warehouse containing nearly 600,000 rocks. Your goal is to pick out the gems as quickly as possible without having to inspect every single rock one by one. This is essentially what Virtual Screening does in drug discovery: it tries to find the few "active" molecules that might cure a disease among millions of "inactive" ones.

This paper is a report card on four different "search strategies" (tools) the researchers used to find these gems in a very realistic, messy warehouse called LIT-PCBA. Unlike previous tests that used fake, easy-to-spot gems, this warehouse contains real rocks and real gems, making the job much harder.

Here is a breakdown of the tools they tested and what they found, using simple analogies:

The Four Search Strategies

1. AutoDock (The Old-School Surveyor):

   • How it works: This is a traditional, physics-based tool. It's like a surveyor who uses a map and a compass to calculate exactly where a rock fits in a hole. It's slow but relies on the laws of physics.
   • The Test: They ran this 10 times for every rock to find the best fit.

2. DiffDock (The AI Intuitionist):

   • How it works: This is a brand-new, AI-powered tool. Instead of calculating physics, it uses "intuition" (a diffusion model) to guess where a rock might go, similar to how a human might guess where a puzzle piece fits without measuring it. It was supposed to be the future of finding gems.
   • The Test: It generated 20 guesses for every rock and picked the one it felt most confident about.

3. GNINA (The Expert Appraiser):

   • How it works: This isn't a finder; it's a judge. Once the Surveyor or the Intuitionist finds a spot, GNINA looks at the fit with a "3D camera" (a neural network) to decide if it's a good fit. It's like an expert appraiser who can spot a fake gem that looks real to the naked eye.

4. NMDN (The New Appraiser):

   • How it works: Another expert judge, but trained differently. It was specifically designed to work with the AI Intuitionist (DiffDock).

The Big Experiment

The researchers tried mixing and matching these tools in different ways:

• Solo acts: Just the Surveyor, just the Intuitionist, or the Surveyor/Intuitionist + the Appraisers.
• Teamwork (Consensus): Taking the rankings from multiple tools and averaging them out to see if the group is smarter than the individual.
• The Coach (Machine Learning): They trained a "Coach" (a Machine Learning model) to look at the scores from all the tools and learn how to rank the rocks even better than the tools could on their own.

The Results: Who Found the Gems?

1. The Old-School Surveyor + The Expert Appraiser Won (AutoDock-GNINA)
Surprisingly, the combination of the traditional Surveyor (AutoDock) and the Expert Appraiser (GNINA) was the most reliable single team.

• The Analogy: It was like hiring a careful surveyor to find the spots, and then having a sharp-eyed appraiser double-check them.
• The Score: They found about 2.14 times more gems in the top 1% of rocks than if you had just picked rocks at random.
• The Catch: Even the winner was only slightly better than random chance. If you picked 100 rocks, you'd find about 2 gems. That's not great, but it's better than nothing.

2. The AI Intuitionist (DiffDock) Struggled
The new, high-tech AI tool (DiffDock) did not perform as well as hoped.

• The Analogy: The AI was great at guessing where things might go, but when the Expert Appraiser (GNINA) looked at those guesses, they were often wrong.
• The Score: It found fewer gems than the old-school Surveyor. In some cases, it found zero gems in the top 1%.
• The Cost: The AI tool was also 4 to 8 times more expensive (in terms of computer power and time) to run than the Surveyor. It was like hiring a celebrity chef to cook a simple sandwich when a home cook could do it faster and cheaper.

3. The New Appraiser (NMDN) Was Inconsistent
When they tried to use the NMDN judge with the AI Intuitionist, it didn't help much. In fact, it sometimes made things worse. It seemed to only work well on specific types of rocks, not all of them.

4. The "Coach" (Machine Learning) Was the Real MVP
The biggest surprise was the Machine Learning model.

• The Analogy: Imagine taking all the notes from the Surveyor, the Intuitionist, and the Appraisers, and feeding them into a super-smart Coach who learned the patterns of what makes a "winning" rock.
• The Score: This Coach doubled the success rate! It went from finding 2 gems per 100 rocks to finding 4.5 gems.
• The Warning: This only worked because they had a lot of data to train the Coach. The paper warns that if you try to use this Coach on a completely new type of rock (a new disease target) without training data, it might fail.

Key Takeaways in Plain English

• No Magic Bullet: There is no single tool that works perfectly for every target. Sometimes the old-school method works best; sometimes the AI works better. It depends on the specific "warehouse" (protein target).
• Newer isn't always better: The fancy AI docking tool (DiffDock) was slower, more expensive, and less accurate than the traditional method when tested on this realistic dataset.
• Teamwork helps, but doesn't solve everything: Combining different tools (Consensus) made the results more stable and reliable, but it didn't beat the best single team (AutoDock-GNINA).
• The Human (or AI) Coach is powerful: If you have enough data, a Machine Learning model that learns from the results of other tools can significantly improve your chances of finding the right drug.
• The Reality Check: Even the best method in this study was only slightly better than random guessing. This means that while these tools are useful for narrowing down the list, they are not perfect. You still have to do a lot of real-world testing to find the actual cures.

In short: The researchers found that a reliable, traditional approach combined with a smart appraiser is currently the best "bang for your buck," but a smart AI Coach can double your success if you have the data to train it. However, the whole process is still far from perfect, and finding a drug remains a difficult, hit-or-miss game.

Technical Summary

Technical Summary: Benchmarking Single-Pose Docking, Consensus Rescoring, and Supervised ML on the LIT-PCBA Library

Problem Statement

Virtual screening (VS) is a critical component of modern drug discovery, aiming to computationally prioritize compounds for experimental testing. However, the predictive power of VS methods is often compromised by the use of biased benchmarking datasets (e.g., DUD-E) that contain analog biases and decoy artifacts, leading to inflated performance metrics. Furthermore, while recent Artificial Intelligence (AI) methods like DiffDock report strong results on standard benchmarks, their practical utility on experimentally curated, unbiased libraries remains uncertain. There is a need to rigorously evaluate whether state-of-the-art AI-driven docking and rescoring methods outperform traditional physics-based approaches on realistic datasets characterized by low hit rates, and whether consensus strategies or supervised machine learning (ML) can further improve early enrichment.

Methodology

The study evaluated docking and scoring workflows on the LIT-PCBA (Literature-derived PubChem Bioassay) library, comprising 578,295 ligand–target pairs across 15 diverse protein targets (10,008 actives and 568,287 inactives). The evaluation focused on two primary pathways for pose generation:

1. AutoDock Pathway: Utilized AutoDock-GPU (a GPU-accelerated implementation of AutoDock 4.2). Each ligand underwent 10 independent runs, with the best-affinity pose retained.
2. DiffDock Pathway: Utilized DiffDock, a diffusion generative model for blind docking. 20 poses were sampled per ligand, with the highest-confidence pose retained.

Scoring and Rescoring:
Poses from both pathways were rescored using:

• GNINA: A 3D convolutional neural network (CNN) scoring function.
• NMDN (Normalized Mixture Density Network): A deep learning-based rescoring function predicting pKd values.
• Native Scores: AutoDock's binding energy and DiffDock's internal confidence scores.

Consensus and Filtering Strategies:
The study implemented rank-based consensus scoring to combine multiple methods:

• Unweighted Consensus: Simple averaging of ranks.
• Calibrated Consensus (CC): Weighted averaging where ML-based scorers (GNINA, NMDN) were given double weight (2×) compared to the baseline docking score (1×).
• Filtering: Three filtering strategies (Medium, Strong, Weak) were applied based on score cutoffs (e.g., NMDN score ≥ -800, GNINA CNNaffinity ≥ 0.1) before ranking.
• Global Consensus: Merged rankings from both AutoDock and DiffDock pathways.

Supervised Machine Learning:
Several ML models (Wide Neural Networks, Deep MLPs, Random Forests, XGBoost, LightGBM) were trained to re-rank ligands. Features included raw docking scores, derived non-linear transforms, and interaction terms. Models were trained on a 75/25 split of data per target to prevent data leakage.

Metrics:
Performance was assessed using Enrichment Factor at 1% and 10% (EF1%, EF10%), ROC-AUC, BEDROC (α=20), and classical classification metrics (Precision, Recall, Balanced Accuracy).

Key Results

1. Single-Method Performance

• AutoDock-GNINA emerged as the most dependable single method, achieving a median EF1% of 2.14 and a balanced accuracy of 50.5%. It significantly outperformed DiffDock-based methods and baselines.
• DiffDock Limitations: DiffDock alone (median EF1% = 1.17) and DiffDock rescored with GNINA (median EF1% = 0.84) or NMDN (median EF1% = 0.67) performed poorly. Notably, DiffDock-based variants failed completely (EF1% = 0) on the OPRK1 target, whereas AutoDock-GNINA achieved an EF1% of 12.5 on the same target.
• NMDN Inconsistency: NMDN rescoring was inconsistent. While it improved performance on specific targets (e.g., FEN1 when paired with AutoDock poses), it often degraded performance when applied to DiffDock poses or other targets.

2. Consensus and Filtering

• Robustness: Calibrated consensus ranking (CC-Medium) improved robustness, yielding a median EF1% of 1.8 in both pathways. It successfully rescued two targets (FEN1 and IDH1) that AutoDock-GNINA alone missed.
• Global Consensus: Merging both pathways via Global Consensus achieved a median EF1% of 1.9, the highest among consensus variants, but still did not surpass the performance of AutoDock-GNINA alone.
• Filtering Trade-offs: Strong filtering (UC-Strong) produced high average EF1% on specific outliers (e.g., PPARG) but resulted in a median EF1% of 0 and near-zero active retention, indicating high volatility. Weak filtering (CC-Weak) maintained high active recovery (~100%) with competitive enrichment.

3. Supervised Machine Learning

• Significant Gains: Supervised ML re-ranking delivered the largest improvements. The best model, a Wide Neural Network (WNN), achieved an EF1% of 4.49, representing a ~110% improvement over the best classical scorer (AutoDock-GNINA at 2.14).
• Metric Improvements: The WNN improved balanced accuracy from 50.5% (AutoDock-GNINA) to 55.4%, precision from 1.85% to 8.2%, and recall from 2.02% to 15.0%.
• Comparison: ML models outperformed all rank-based consensus strategies (median EF1% ~1.8–1.9), suggesting they can capture non-linear patterns across scoring functions that simple averaging cannot.

4. Computational Efficiency

• Throughput: AutoDock was significantly more cost-effective and faster than DiffDock. AutoDock processes approximately 2 ligands/second on an RTX 3090, while DiffDock requires 1–2 seconds per ligand on a more powerful A100 GPU. The study estimates DiffDock requires 4 to 8 times the computational power to deliver inferior precision in this context.

Significance and Claims

The paper concludes that while docking-based virtual screening provides useful enrichment, its absolute predictive power on realistic datasets like LIT-PCBA remains limited. Key claims include:

1. No Universal Solver: No single docking technique works reliably across all targets. Performance is highly target-dependent, with methods like DiffDock failing completely on specific targets (e.g., OPRK1) where classical methods (AutoDock-GNINA) succeed.
2. Superiority of Classical + ML Rescoring: The combination of a traditional physics-based sampler (AutoDock) with a modern ML rescoring function (GNINA) offers the best balance of accuracy, cost, and reliability among single methods.
3. Value of Supervised Learning: Supervised ML models can significantly boost enrichment (doubling EF1%) if sufficient experimental data is available for training, outperforming both single scorers and simple consensus strategies.
4. Caution Against Novelty: The study advocates for employing rigorously tested, well-understood docking techniques over relying on unpredictable novel AI methods that may yield inconsistent results, particularly when training data biases (e.g., "near neighbors" in DiffDock's training set) may compromise generalizability.
5. Inherent Limitations: Even the best-performing methods (AutoDock-GNINA, ML re-rankers) only achieve modest discrimination (Balanced Accuracy ~50–55%), suggesting that structure-based docking alone cannot fully solve the hit-finding problem and may be constrained by factors beyond sampling and scoring.

The authors emphasize that effective workflows should prioritize methods validated against specific targets or closely related systems rather than assuming universal superiority of new AI-driven approaches.