Assessing the potential of deep learning for protein-ligand docking

This paper introduces PoseBench, the first comprehensive benchmark designed to systematically evaluate deep learning methods for protein-ligand docking under challenging real-world conditions, including the use of predicted apo structures, concurrent multi-ligand binding, and unknown binding pockets, revealing that while co-folding methods generally outperform baselines, they still struggle with novel poses and balancing structural accuracy with chemical specificity.

The Gist

Imagine you are a master locksmith trying to open a complex, high-tech safe. The "safe" is a protein (a tiny machine inside our bodies), and the "key" is a drug molecule (a ligand). To cure a disease, you need to know exactly how that key fits into the lock. If the key is even a millimeter off, the door won't open, or worse, it might jam the mechanism.

For decades, scientists have tried to predict this fit using computers. Recently, Deep Learning (AI) has become the new superstar in this field, promising to solve these puzzles faster than ever before. But here's the catch: How do we know if these AI lockpickers are actually good, or if they just got lucky on the practice locks they studied?

This paper introduces PoseBench, a giant, rigorous "final exam" for these AI models to see if they can really handle real-world drug discovery.

The Problem: The "Practice Test" Trap

Imagine a student who memorizes the answers to a specific practice test. They get 100% on the practice test, but when they face a new test with slightly different questions, they fail.

Many AI models for protein docking have been trained on old data (like the "Practice Test"). They are great at predicting how a drug fits into a protein they've seen before. But in the real world, scientists need to design drugs for brand new proteins (proteins we've never seen), multiple drugs at once (like a team of keys working together), or proteins where we don't even know where the lock is yet.

The authors asked: Do these AI models actually understand the rules of chemistry, or are they just memorizing patterns?

The Solution: PoseBench (The Ultimate Obstacle Course)

The authors built PoseBench, a comprehensive benchmark (a standardized test) that puts AI models through three specific, difficult challenges:

1. The "Blind Date" Challenge (No Map):

   • Analogy: Imagine trying to find a specific room in a giant, dark mansion without a floor plan. You have to guess where the door is.
   • The Test: The AI is given a protein it has never seen before and must find the "binding pocket" (the lock) without being told where it is.
   • Result: Some AI models (like AlphaFold 3) are amazing at this, but only if they have a "family tree" (evolutionary data) to help them. Others struggle to find the door at all.

2. The "Group Hug" Challenge (Multiple Keys):

   • Analogy: Imagine trying to fit three different keys into one lock at the same time without them bumping into each other.
   • The Test: Can the AI predict how multiple drug molecules interact with a protein simultaneously?
   • Result: This is very hard. Most AI models get confused and make the keys crash into each other (a "steric clash"). Only the most advanced models can manage a clean fit.

3. The "New Neighborhood" Challenge (Out-of-Distribution):

   • Analogy: The student takes a test in a completely different city with different traffic rules.
   • The Test: The AI is tested on proteins that are totally different from anything it saw during training (like immune system proteins or metal-transporting proteins).
   • Result: The AI models often fail here. They are "overfitted" to common proteins and don't know how to handle the weird, rare ones.

The Key Findings (The Report Card)

• The Winners: The new "Co-folding" models (like AlphaFold 3, Chai-1, and Boltz-1) are generally the best. They don't just look at the lock; they imagine the whole key and lock changing shape to fit each other. They beat the old-school methods significantly.
• The Catch: AlphaFold 3 is the current champion, but it has a secret weakness. It relies heavily on having a "Multiple Sequence Alignment" (MSA)—basically, a huge family tree of similar proteins. If you take that family tree away, AlphaFold 3's performance drops drastically. It's like a detective who can only solve crimes if they have a witness; without the witness, they are lost.
• The Flaw: Even the best AI models struggle with chemical specificity. They might get the shape of the key right (structural accuracy), but they might get the "chemistry" wrong (e.g., the key is made of the wrong material and won't turn). They are good at geometry but sometimes bad at chemistry.
• The "Black Holes": The AI models consistently fail to predict how drugs interact with immune system proteins and metal-transporting proteins. These are the "blind spots" in their training data.

Why This Matters

This paper is a reality check. It tells us that while AI is a powerful tool for drug discovery, it isn't magic yet.

• For Scientists: Don't just trust the AI blindly. If you are designing a drug for a rare protein or a complex immune system target, the AI might be hallucinating the fit. You need to double-check with experiments.
• For the Future: The authors suggest that future AI needs to be trained on more diverse data (especially those "weird" proteins) and needs to learn the rules of chemistry better, not just the shapes.

The Bottom Line

Think of PoseBench as the "Olympics" for protein-docking AI. The current champions (like AlphaFold 3) are incredible athletes, but they still trip over the hurdles of rare proteins and complex chemical interactions. This benchmark gives scientists a clear map of where the AI is strong and where it needs more training before we can fully trust it to design life-saving medicines.

The code and data are now open for everyone to use, so the whole scientific community can help build the next generation of AI that can truly master the art of the molecular lockpick.

Technical Summary

1. Problem Statement

The field of drug discovery relies heavily on accurately predicting the 3D structure of protein-ligand complexes (Protein-Ligand Interactions, or PLIs). While Deep Learning (DL) has revolutionized protein structure prediction (e.g., AlphaFold 2), its application to protein-ligand docking faces significant gaps in real-world applicability. Existing benchmarks and methods often fail to address three critical, practical scenarios:

1. Apo-to-Holo Docking: Using predicted protein structures (apo) rather than experimentally determined crystal structures (holo) for docking, which is necessary for novel proteins.
2. Multi-Ligand Binding: Simultaneously predicting the binding poses of multiple ligands (e.g., cofactors) to a single target, crucial for enzyme design.
3. Blind Docking: Predicting binding poses without prior knowledge of the binding pocket location.

Furthermore, current DL methods often struggle to balance structural accuracy (correct 3D pose) with chemical specificity (correct amino acid-specific interactions), particularly for novel or uncommon protein targets.

2. Methodology: PoseBench

To address these gaps, the authors introduced PoseBench, the first comprehensive benchmark designed to systematically evaluate DL docking and co-folding methods under broadly applicable, realistic conditions.

A. Benchmark Datasets

PoseBench utilizes four distinct datasets to test varying levels of difficulty and novelty:

1. Astex Diverse (Easy): 85 complexes (deposited pre-2007). Represents "in-distribution" data likely present in training sets.
2. PoseBusters Benchmark (Intermediate): 308 complexes (deposited post-2019). Tests generalization to newer data.
3. DockGen-E (Challenging): 122 complexes with functionally distinct binding pockets (ECOD domains). Designed to test performance on uncommon binding sites.
4. CASP15 (Challenging/Multi-Ligand): 19 complexes (102 ligands) released post-2022. Includes both single and multi-ligand targets, strictly out-of-distribution for all baseline models.

Input Standardization: Crucially, all experiments use predicted apo-protein structures (generated via AlphaFold 3 or ESMFold) and standardized Multiple Sequence Alignments (MSAs), rather than crystal structures, to simulate real-world discovery workflows.

B. Baseline Methods Evaluated

The benchmark compares three categories of algorithms:

• Conventional: P2Rank (binding site prediction) + AutoDock Vina (docking).
• DL Docking: DiffDock-L, DynamicBind, NeuralPLexer (generative approaches).
• DL Co-folding: RoseTTAFold-All-Atom (RFAA), Chai-1, Boltz-1, and AlphaFold 3 (AF3).
• Note: Both full MSA and "Single-Sequence" (MSA-ablated) versions of co-folding models were tested to assess MSA dependency.

C. Evaluation Metrics

The authors introduced new metrics to evaluate chemical specificity alongside structural accuracy:

• Structural Accuracy: Root Mean Square Deviation (RMSD) and centroid RMSD (cRMSD).
• Chemical Validity: PoseBusters (PB-Valid): A suite checking for steric clashes, energy ratios, and geometric sanity.
• Interaction Specificity (New):
  • PLIF-EMD: Earth Mover's Distance between predicted and native Protein-Ligand Interaction Fingerprint (PLIF) histograms.
  • PLIF-WM: A normalized Wasserstein Matching score (0–1) derived from PLIF-EMD, measuring how well a model recapitulates the distribution of specific amino acid-ligand interactions.

3. Key Contributions

1. PoseBench Framework: A unified, open-source toolkit for evaluating protein-ligand docking using predicted apo-structures and blind docking protocols.
2. Multi-Ligand Benchmarking: The first systematic evaluation of DL methods on multi-ligand complexes (CASP15), revealing significant challenges in modeling inter-ligand steric clashes.
3. New Metrics: Introduction of PLIF-WM to quantify the chemical specificity of predicted interactions, addressing the limitation of RMSD-only evaluations.
4. Systematic Analysis of Failure Modes: Identification of specific protein classes (e.g., immune system proteins, metal transporters) where current DL models consistently fail.

4. Key Results

A. Performance Hierarchy

• DL Co-folding vs. Conventional: DL co-folding methods (AF3, Chai-1, Boltz-1) generally outperform conventional docking (P2Rank-Vina) and DL docking baselines in structural accuracy and pocket identification.
• The "Novelty" Gap: While DL methods excel on the Astex Diverse dataset (high overlap with training data), performance drops significantly on DockGen-E and CASP15. Even AlphaFold 3 fails to achieve >50% success on DockGen-E for structurally and chemically accurate poses.

B. MSA Dependency

• High Sensitivity: AlphaFold 3 (AF3) is heavily dependent on diverse input MSAs. Its single-sequence variant shows catastrophic performance degradation.
• Robustness: Chai-1 and Boltz-1 are less sensitive to MSA availability, with their single-sequence variants performing comparably to their MSA-based versions on certain datasets.

C. Structural Accuracy vs. Chemical Specificity Trade-off

• The Trade-off: Methods that achieve high structural accuracy (low RMSD) often fail to predict the correct chemical interactions (low PLIF-WM).
• AF3's Limitation: While AF3 achieves top structural accuracy on CASP15 multi-ligand targets (with MSAs), it frequently produces chemically implausible poses (low PB-Valid scores), hallucinating interactions not native to the protein.
• Boltz-1's Balance: Boltz-1 often strikes a better balance between structural accuracy and chemical specificity, even without MSAs, suggesting better generalization for certain tasks.

D. Failure Modes

Analysis of failed predictions reveals that DL models struggle most with:

• Immune system proteins
• Metal transport proteins
• Biosynthetic proteins
• RNA-binding proteins
• Oxidoreductases
  These classes are underrepresented in current training data (PDB), leading to poor generalization.

5. Significance and Future Directions

• Real-World Utility: The study demonstrates that while DL co-folding is a powerful tool, it is not yet a "plug-and-play" solution for novel drug discovery, particularly for proteins with uncommon binding pockets or multi-ligand requirements.
• Data Bias: The results highlight a critical bias in current training data (PDB), where models are "overfitted" to common interaction types and fail on rare but biologically important classes.
• Future Research:
  • Development of physico-chemical loss functions to better balance structural and chemical accuracy.
  • Fine-tuning strategies for uncommon protein classes (e.g., immune system).
  • Retraining generative models directly on multi-ligand complexes to resolve steric clash issues.
  • Investigating the role of protein language model embeddings vs. MSAs in reducing data dependency.

Conclusion: PoseBench establishes that while DL has surpassed conventional methods in many docking tasks, significant challenges remain in generalizing to novel targets, handling multi-ligand systems, and ensuring chemical validity. The benchmark serves as a critical resource for guiding the next generation of biomolecular structure prediction tools.

Code, data, and tutorials are available at: https://github.com/BioinfoMachineLearning/PoseBench