On the Reliability of AI Methods in Drug Discovery: Evaluation of Boltz-2 for Structure and Binding Affinity Prediction

Here is an explanation of the paper, translated into simple language with some creative analogies.

The Big Picture: The "Magic Crystal Ball" vs. The "Hard-Working Scientist"

Imagine you are trying to find a specific key (a drug) that fits into a very complex, locked door (a virus or cancer cell).

For a long time, scientists have used Physics-Based Methods (like the ESMACS protocol mentioned in the paper). Think of this as a Master Locksmith. The locksmith doesn't guess; they physically try thousands of keys, feeling the tumblers click, measuring the exact pressure, and calculating the energy required to turn the lock. It's incredibly accurate, but it's slow, expensive, and requires heavy machinery.

Recently, a new tool called Boltz-2 arrived. It's an AI "Magic Crystal Ball." It has read millions of photos of keys and locks. When you show it a new key, it instantly guesses what it looks like and how well it might fit, without ever touching the lock. It's lightning fast and cheap.

The Question: Can we trust the Magic Crystal Ball to do the job of the Master Locksmith?

The Experiment: Putting Boltz-2 to the Test

The authors of this paper decided to put Boltz-2 through a massive stress test. They didn't just look at a few keys; they tested it on two different types of locks (proteins called 3CLPro and TNKS2) using over 38,000 different keys (drug compounds).

They compared Boltz-2's "guesses" against the Master Locksmith's (ESMACS) "measurements."

What They Found: The Good, The Bad, and The Weird

1. The "Where is the Key?" Problem (Structure Prediction)

The Locksmith: Always puts the key in the exact right hole.
The Crystal Ball: Sometimes puts the key in the right hole, but often puts it in a completely different part of the lock, or even on the outside of the door.
The Analogy: Imagine asking an AI to draw a picture of a cat sitting in a chair. Sometimes it draws a cat in the chair. Other times, it draws a cat floating in the ceiling or a chair made of jelly.
The Result: For one of the proteins (TNKS2), the AI was okay at finding the right "chair." For the other (3CLPro), it was very confused, often placing the drug in the wrong spot entirely.

2. The "How Good is the Fit?" Problem (Binding Affinity)

The Locksmith: Can tell you exactly how hard it is to turn the key (e.g., "This key needs 5 Newtons of force").
The Crystal Ball: It tries to guess the "force," but it has a weird habit of saying "Everything is about the same."
The Analogy: Imagine a weather app that predicts rain. When it's actually a hurricane, it says "Light drizzle." When it's a sunny day, it also says "Light drizzle." It can't tell the difference between a storm and a breeze.
The Result: Boltz-2 predicted that almost all drugs were "moderately good." It couldn't distinguish between a "super drug" and a "useless drug." When the researchers looked at the top 100 best guesses from the AI, they found zero correlation with the real physics measurements. The AI's "best" picks were often just random noise compared to the real data.

3. The "Chemical Glitch" (Saturation Errors)

The Weirdness: When the AI drew the 3D structure of the drug, it sometimes got the chemistry wrong.
The Analogy: Imagine the AI is drawing a car. It gets the shape right, but it accidentally turns the rubber tires into steel wheels, or the glass windows into solid metal. It looks like a car, but it's chemically impossible.
The Result: The AI sometimes predicted that a flexible part of a drug was rigid, or that a ring of atoms was missing hydrogen atoms. These small chemical errors completely change how a drug works, making the AI's predictions unreliable for the final stage of drug discovery.

The Verdict: A Great Scout, But a Bad Sniper

The paper concludes that Boltz-2 is a fantastic "Scout," but a terrible "Sniper."

As a Scout (Early Screening): If you have a billion drugs and need to quickly throw away the obvious junk, Boltz-2 is amazing. It's fast and can give you a rough idea of what might work.
As a Sniper (Lead Identification): If you need to find the one perfect drug to save a life, you cannot trust Boltz-2 alone. It lacks the "resolution" to see the tiny, critical differences between a good drug and a great one.

The Takeaway

The authors argue that we cannot simply replace the "Master Locksmith" (Physics) with the "Magic Crystal Ball" (AI).

AI is great at spotting patterns, but drug discovery is full of "Activity Cliffs"—tiny changes in a molecule that cause massive changes in how it works. AI, which relies on smooth patterns, often misses these cliffs.

The Solution? We need to use AI to do the heavy lifting and speed things up, but we must still use the "Master Locksmith" (Physics-based simulations) to double-check the most important candidates. We need the speed of AI combined with the precision of physics to truly revolutionize drug discovery.

In short: Don't let the AI drive the car all the way to the hospital; let it navigate the highway, but make sure a human (or a physics engine) is ready to take the wheel for the final, critical turn.

Here is a detailed technical summary of the paper "On the Reliability of AI Methods in Drug Discovery: Evaluation of Boltz-2 for Structure and Binding Affinity Prediction."

1. Problem Statement

Despite the rapid advancement of Artificial Intelligence (AI) in drug discovery, no AI-discovered drug has yet received regulatory approval. A critical gap exists between the speed of AI methods and the physical accuracy required for reliable lead identification. Specifically, while "foundation models" like Boltz-2 claim to bridge this gap by predicting protein-ligand structures and binding affinities with high fidelity (approaching physics-based methods), their reliability in large-scale, target-specific screening remains unverified. The core challenge is determining whether these AI models can accurately quantify interaction energetics and predict binding poses without the rigorous sampling and uncertainty quantification provided by physics-based simulations.

2. Methodology

The authors conducted a comprehensive evaluation of Boltz-2, a billion-parameter biomolecular foundation model, against established physics-based benchmarks.

Datasets: Two large-scale, therapeutically relevant datasets were used:
- 3CLPro (SARS-CoV-2 main protease): 16,780 compounds.
- TNKS2 (Tankyrase 2): 21,702 compounds.
- These compounds were generated via iterative active learning (REINVENT) and surrogate docking.
AI Model (Boltz-2):
- Utilized a "co-folding" architecture to predict protein-ligand complexes and binding affinities simultaneously without spatial constraints.
- Inference was performed on NVIDIA GH200 GPUs (Isambard-AI).
Physics-Based Benchmark (ESMACS):
- Global Scale: Used a coarse-grained ESMACS (Enhanced Sampling of Molecular Dynamics with Approximation of Continuum Solvent) protocol (10 replicas, 4ns) to generate binding free energies ( $\Delta G$ ) for the full datasets.
- High-Precision Subset: For the top 100 compounds ranked by Boltz-2, a fine-grained FG-ESMACS protocol was applied (25 replicas, 10ns) on the Frontier exascale supercomputer to provide high-resolution, reproducible binding free energies with statistical uncertainty quantification.
Comparative Metrics:
- Structural: Root Mean Square Deviation (RMSD) for global pose and Local Distance Difference Test (LDDT) for local binding site fidelity.
- Energetic: Pearson correlation ( $r$ ) and Spearman rank correlation ( $\rho$ ) between Boltz-2 predicted affinities and ESMACS $\Delta G$ .
- Chemical Integrity: Analysis of saturation states and protonation consistency between Boltz-2 predictions and canonical SMILES strings.

3. Key Contributions

Large-Scale Benchmarking: This is one of the first rigorous evaluations of a foundation model (Boltz-2) on datasets exceeding 16,000–21,000 compounds, moving beyond small, curated test sets.
Dual-Validation Approach: The study uniquely compares AI predictions against both traditional docking and high-precision, ensemble-based physics simulations (ESMACS), highlighting the discrepancy between AI speed and physical accuracy.
Identification of Systemic Artifacts: The authors identified specific chemical artifacts in AI-generated structures, such as incorrect saturation states (e.g., predicting aromatic rings where saturated rings exist) and protonation errors, which fundamentally alter binding energetics.
Uncertainty Quantification Contrast: The paper contrasts the "overconfident" nature of AI confidence scores (which lack discrimination at high thresholds) with the rigorous statistical uncertainty provided by ensemble physics methods.

4. Key Results

A. Structural Fidelity

Protein Conformations: Boltz-2 predicted a single, accurate protein conformation for 3CLPro (RMSD $\approx$ 0.4 Å) but showed significant variation for TNKS2, predicting multiple conformations rather than a single native state.
Ligand Poses:
- 3CLPro: A significant portion of ligands were predicted to bind in sites entirely different from the crystallographic binding site (high RMSD and LDDT).
- TNKS2: While most ligands bound near the correct site, many exhibited incorrect orientations (rotated/flipped) within the pocket.
- Confidence Scores: Boltz-2 confidence scores were uniformly high (>0.8 for all ligands) and compressed, offering little discriminatory power to filter out poor predictions.

B. Binding Affinity Predictions

Global Correlation: Weak to moderate correlations were observed between Boltz-2 predictions and ESMACS energies ( $r \approx 0.24$ for 3CLPro, $r \approx 0.45$ for TNKS2).
"Regression to the Center": Boltz-2 exhibited a tendency to predict affinities within a narrow range (mostly -5 to -8 kcal/mol), failing to distinguish true binders from non-binders (decoys).
Top-100 Analysis (Critical Failure): When analyzing the top 100 compounds ranked by Boltz-2:
- Zero Correlation: There was no significant correlation between Boltz-2 predictions and the high-precision FG-ESMACS results ( $r \approx 0$ ).
- Chemical Artifacts: A systematic analysis revealed that Boltz-2 often altered the chemical identity of ligands (e.g., changing saturation states of rings and chains). Correcting these chemical states did not improve the correlation, indicating the model's affinity predictions are decoupled from the actual physical pose.
- Overlap: The top 100 compounds identified by Boltz-2 and ESMACS had almost no overlap (1-2 compounds), suggesting the models explore fundamentally different regions of chemical space.

5. Significance and Conclusion

Limitations of Current AI: The study concludes that while Boltz-2 offers substantial speed for initial screening, it lacks the energetic resolution required for lead identification and optimization. It cannot replace physics-based methods for ranking compounds in the hit-to-lead phase.
Root Cause: The failure stems from the assumption that biomolecular data behaves as a smooth, differentiable manifold. AI models rely on statistical patterns from training data and fail to capture "activity cliffs" (non-linear changes in affinity due to minor structural changes) and the discontinuous nature of the physical energy landscape.
Recommendation: The authors argue that AI-derived models must be rigorously validated and refined using physics-based methods. Foundation models currently rely on "ad hoc statistical heuristics" that are decoupled from local atomic environments.
Future Outlook: Until foundation models can incorporate the rigor and uncertainty quantification of physics-based interactions, they remain insufficient for navigating the complex chemical space of real-world drug discovery. The study underscores the necessity of a hybrid approach where AI accelerates screening, but physics ensures reliability.