PoseBusters: AI-based docking methods fail to generate physically valid poses or generalise to novel sequences

This paper introduces PoseBusters, a validation tool demonstrating that current deep learning-based protein-ligand docking methods often fail to generate physically plausible structures or generalize to novel sequences, thereby underperforming classical docking tools that better incorporate essential physical principles.

The Gist

Imagine you are trying to find the perfect key to fit into a very specific, complex lock. In the world of drug discovery, the "lock" is a protein in your body, and the "key" is a potential medicine (a molecule). The process of figuring out exactly how that key fits into the lock is called docking.

For years, scientists have used traditional, rule-based computer programs to do this. Recently, a new wave of "AI" programs (Deep Learning) has arrived, promising to do the job faster and better. These AI models are like brilliant students who have memorized millions of examples of keys and locks.

However, a new study called PoseBusters suggests that while these AI students are very good at memorizing the shape of the key, they are terrible at understanding the physics of how it actually works.

Here is a simple breakdown of what the paper found:

1. The "RMSD" Trap: Looking Good on Paper

Scientists usually judge how well a docking program works by measuring RMSD. Think of RMSD as a ruler. If the AI predicts where the key goes, and that prediction is within 2 millimeters (Angstroms) of where the key actually sits in a real-life photo (a crystal structure), the AI gets a passing grade.

The paper found that many AI programs get high scores on this ruler test. They say, "Look! We are 90% accurate!"

2. The Reality Check: The "Impossible" Key

The problem is that these AI programs are so focused on matching the ruler measurement that they sometimes create physically impossible keys.

Imagine the AI predicts a key that:

• Has a bond (a connection between atoms) that is stretched so thin it would snap like a dry twig.
• Has a ring shape that is twisted into a pretzel, even though chemistry says it should be flat like a pancake.
• Has two parts of the key crashing into each other like two cars trying to drive through the same door at the same time.

The paper calls these "physically implausible." It's like the AI drew a picture of a key that looks right from a distance, but if you tried to build it, it would fall apart or break the lock.

3. Enter PoseBusters: The Inspector

To catch these bad predictions, the authors built a tool called PoseBusters. Think of PoseBusters as a strict building inspector or a quality control manager.

Instead of just measuring the ruler (RMSD), PoseBusters checks the "laws of physics" for every prediction:

• Chemical Validity: Does the molecule make sense chemically? (e.g., Is the charge correct? Are the atoms connected properly?)
• Geometry: Are the rings flat? Are the bonds the right length?
• Clashes: Did the key crash into the lock or other parts of the machine?

If a prediction fails these checks, it is marked as "invalid," no matter how good the ruler measurement was.

4. The Big Reveal: Old vs. New

The researchers tested five new AI docking methods against two older, traditional methods (AutoDock Vina and Gold).

• On familiar locks (Training Data): When the AI was tested on locks it had seen before during its training, it looked amazing on the ruler test. One AI (DiffDock) seemed to beat the old methods.
• The "Physics" Filter: But when PoseBusters checked the physics, the AI's performance dropped drastically. Many of its "perfect" predictions were actually impossible structures. The old, traditional methods, while slightly slower, produced keys that were both accurate and physically possible.
• On new, unknown locks (Generalization): When the researchers tested the AI on completely new locks it had never seen (a "Benchmark Set"), the AI struggled badly. It couldn't generalize. The old methods, which rely on physical rules rather than just pattern memorization, handled these new locks much better.

5. The "Tweak" Doesn't Fix Everything

The authors tried to help the AI by adding a "polishing" step after the prediction, using a physics engine (called a force field) to smooth out the weird shapes.

• The Result: This helped the AI fix some of its broken keys, but it didn't make them better than the old traditional methods. The old methods were already starting with a solid foundation; the AI had to try to fix a broken foundation.

The Bottom Line

The paper concludes that AI-based docking methods are not yet ready to replace traditional tools.

While they are fast and can guess the right location, they often ignore the basic laws of chemistry and physics. To be truly "state-of-the-art," a method needs to pass two tests:

1. The Ruler Test: Is it in the right spot?
2. The Physics Test: Is it a real, buildable object?

Currently, the traditional methods pass both. The AI methods pass the first but often fail the second. The authors hope that by using their "PoseBusters" tool, developers can fix these AI models to understand physics better, leading to truly accurate drug predictions in the future.

Technical Summary

Problem
Recent years have seen the emergence of numerous deep learning (DL)-based protein-ligand docking methods, which promise significant improvements in speed and accuracy for structure-based drug discovery. However, a critical flaw has been identified: while these methods often report state-of-the-art performance based on the root-mean-square deviation (RMSD) of predicted binding modes relative to crystal structures, they frequently generate physically implausible molecular structures. These structures may violate fundamental chemical rules, such as incorrect bond lengths, distorted aromatic rings, or severe steric clashes between the ligand and the protein. Consequently, evaluating docking methods solely on RMSD is insufficient, particularly for DL-based approaches that may lack the "inductive biases" inherent in classical physics-based methods to ensure chemical consistency and physical realism.

Methodology
To address this gap, the authors introduce PoseBusters, a Python package utilizing the RDKit cheminformatics toolkit to perform a comprehensive suite of quality checks on docking predictions. The test suite validates three categories of properties:

1. Chemical Validity and Consistency: Checks include RDKit sanitization, preservation of molecular formula, bond connectivity, tetrahedral chirality, and double-bond stereochemistry relative to the input ligand.
2. Intramolecular Validity: This group assesses the physical plausibility of the ligand itself, including bond lengths and angles (within 25% of distance geometry bounds), planarity of aromatic rings and double bonds (within 0.25 Å), internal steric clashes, and conformational energy ratios (using the Universal Force Field, UFF).
3. Intermolecular Validity: This group evaluates interactions between the ligand and its environment, checking for protein-ligand and ligand-cofactor clashes based on van der Waals radii (with a 0.75 threshold) and volume overlap (limiting intersection to <7.5%).

The authors applied PoseBusters to evaluate seven docking methods: five DL-based approaches (DeepDock, DiffDock, EquiBind, TankBind, and Uni-Mol) and two established classical methods (AutoDock Vina and CCDC Gold). The evaluation was conducted on two datasets:

• Astex Diverse Set: A standard benchmark containing 85 complexes, many of which overlap with the training data of the DL methods.
• PoseBusters Benchmark Set: A novel, rigorous test set of 308 high-quality complexes released after 2021, ensuring no overlap with the training data (PDBbind General Set v2020) of the tested DL methods. This set was used to assess generalization to novel sequences and structures.

Additionally, the authors performed post-docking energy minimization using molecular mechanics force fields (AMBER ff14sb and Sage) on the DL predictions to determine if physical plausibility could be recovered through classical refinement.

Key Contributions

• PoseBusters Test Suite: The development of an open-source tool that systematically identifies chemically inconsistent and physically implausible ligand poses, moving beyond simple RMSD metrics.
• Rigorous Benchmarking: A comparative analysis demonstrating that current DL-based docking methods fail to generalize to novel sequences and often produce invalid structures, even when achieving low RMSD scores.
• Identification of Missing Physics: Evidence that DL methods lack specific inductive biases regarding chemical geometry and steric constraints, which are naturally encoded in classical force fields.

Results

• Performance on Astex Diverse Set: While DiffDock appeared superior based on RMSD alone (72% of predictions within 2 Å), its performance dropped significantly when physical plausibility was considered (47% PB-valid). Classical methods (Gold and AutoDock Vina) outperformed all DL methods when both RMSD and physical validity were required.
• Performance on PoseBusters Benchmark Set: On this unseen data, the gap widened. Gold and AutoDock Vina remained the top performers. The best DL method, DiffDock, achieved only 12% PB-validity, while other DL methods (EquiBind, Uni-Mol, TankBind) generated almost no physically valid poses.
• Generalization and Overfitting: DL methods performed significantly worse on proteins with low sequence identity (<30%) to their training sets, suggesting severe overfitting to training targets.
• Impact of Energy Minimization: Applying post-docking energy minimization significantly improved the physical plausibility of DL predictions (increasing the number of valid poses), indicating that force fields contain docking-relevant physics missing from current DL models. However, even with this refinement, DL methods did not surpass classical tools.
• Specific Failure Modes: The study identified distinct failure patterns for each DL method, such as TankBind overlooking stereochemistry, Uni-Mol predicting non-standard bond lengths, and EquiBind generating protein-ligand clashes.

Significance
The paper argues that the current standard of evaluating docking methods solely on RMSD is inadequate and misleading. It posits that for a method to be considered "state-of-the-art," it must not only predict native-like binding modes but also generate chemically consistent and physically plausible structures. The authors claim that PoseBusters provides a necessary framework for identifying missing inductive biases in deep learning models. By highlighting that current DL methods do not yet outperform classical docking tools when physical realism is accounted for, the work aims to drive the development of more accurate and realistic prediction methods that integrate the speed of deep learning with the physical rigor of classical force fields.