Assessing the Generalizability of Machine Learning and Physics Methods for DNA-Encoded Libraries

This study evaluates the generalizability of machine learning and physics-based methods for DNA-encoded library screening, revealing that while ML excels in-distribution, optimal out-of-distribution hit discrimination is target- and ligand-dependent, thereby necessitating rigorous system-specific pilot testing and providing the open-source DEL-iver toolkit to support these workflows.

The Gist

The Big Picture: Finding a Needle in a Billion-Haystack

Imagine you are a treasure hunter looking for a specific type of gold coin (a drug) that can fix a broken machine (a disease).

DNA-Encoded Libraries (DELs) are like a massive warehouse containing billions of different coins. To find the right one, you dump the whole pile into a machine that only lets the "good" coins stick to a magnet. You then count how many of each coin stuck. This is great, but the coins in the warehouse are made of a very specific, limited set of Lego bricks. They are all built the same way.

The big question scientists asked was: "Can we teach a computer to look at the coins we found in the warehouse and predict which new coins (made of different Legos) will also stick to the magnet?"

This is like trying to teach a dog to find a specific ball in a park, and then expecting that same dog to find a different ball in a completely different forest without ever seeing that forest before.

The Problem: The "NeurIPS" Challenge

Recently, a huge competition (like a Super Bowl for AI) called BELKA tried to solve this. They gave 2,000 teams of data scientists the "warehouse" data and asked them to predict the winners in a "forest" (new, unseen chemicals).

The Result? Everyone failed. Even the best AI models could guess well if the new coins looked exactly like the old ones, but as soon as the coins were slightly different, the AI got confused. It was like the dog only knew how to find the red ball, but when you gave it a blue ball, it didn't know what to do.

What This Paper Did: The "Detective" Work

The authors of this paper decided to investigate why the AI failed and if they could fix it by mixing in some "physics" (how molecules actually move and stick together). They acted like detectives testing three different tools:

1. The "Pattern Matcher" (Machine Learning): This is the AI that just looks at the chemical names and guesses based on patterns it saw before.
2. The "3D Simulator" (Docking): This is a physics-based tool that builds a 3D model of the coin and the magnet to see if they physically fit together.
3. The "Hybrid" (Co-folding): A fancy new AI that tries to simulate the coin and magnet folding together in real-time.

The Key Findings (The "Aha!" Moments)

Here is what they discovered, translated into everyday terms:

1. The "Same Recipe" Rule

If the new coins are made with the same Lego bricks as the old ones, just arranged differently, the "Pattern Matcher" (AI) is a genius. It works perfectly.

• The Catch: If the new coins use brand new Lego bricks the AI has never seen, the AI is useless. It's like trying to guess the flavor of a new fruit just by knowing how apples and oranges taste.

2. The "Garbage In, Garbage Out" Myth

DEL data is messy. For every 100 coins, 99 are junk (non-binders) and only 1 is a winner. The team thought they needed all 100 to train the AI.

• The Surprise: They threw away 90% of the junk coins and trained the AI on the remaining 10. The AI performed just as well!
• The Lesson: You don't need a massive library of junk to teach the AI; you just need a clean, high-quality set of examples. It's like teaching a kid to recognize a cat; you don't need to show them 1,000 pictures of rocks to prove it's not a cat.

3. The "One Size Does Not Fit All"

This is the most important finding. The team tried the "3D Simulator" and "Hybrid" tools to help the AI guess the new coins.

• Target A (BRD4): The "Hybrid" tool (Boltz-2) was a superhero. It could find the right coins in the forest when the AI failed.
• Target B (sEH): The "Hybrid" tool was confused. But the "3D Simulator" (GALigandDock) was the superhero here.
• The Lesson: There is no single "magic wand." What works for one disease might fail for another. You have to test your tools on a small scale before you bet the farm on them.

The Solution: "DEL-iver"

To help other scientists avoid these pitfalls, the authors built a free, open-source toolbox called DEL-iver.

Think of DEL-iver as a Swiss Army Knife for drug hunters. Instead of needing a PhD in computer science to run these complex tests, anyone can use this tool to:

• Clean up their messy data.
• Test if their AI model is actually good at guessing new chemicals.
• Mix in physics simulations to get better results.
• Visualize the results easily.

The Bottom Line

The paper concludes that while Artificial Intelligence is powerful, it has a blind spot: it struggles to predict things it has never seen before.

To find new medicines, we can't just rely on the AI's "gut feeling." We need to:

1. Test small first: Run a "pilot test" to see if the method works for your specific target.
2. Mix methods: Sometimes you need the AI, sometimes you need the physics simulator, and sometimes you need both.
3. Don't trust the hype: Just because a model works on a leaderboard doesn't mean it will work in the real world.

By using their new toolbox (DEL-iver), scientists can stop guessing and start making reliable predictions about which new drugs might actually work.

Technical Summary

1. Problem Statement

DNA-encoded libraries (DELs) enable the ultra-high-throughput screening of billions of molecules. However, the chemical space of DELs is often limited by specific synthesis constraints (e.g., specific scaffolds and building blocks). A primary goal in the field is to train Machine Learning (ML) models on DEL data to predict the activity of "off-DNA" compounds (purchasable analogues) that lie outside the training distribution (Out-of-Distribution or OOD).

The NeurIPS 2024 BELKA competition highlighted a critical failure in this approach: despite access to ~133 million data points, no competing team successfully trained a model that could generalize to OOD chemical space. The top-performing models achieved a Mean Average Precision (mAP) of only 0.36 on the private test set, indicating that current ML methods struggle to extrapolate beyond the specific building blocks and scaffolds seen during training.

2. Methodology

The authors systematically evaluated the performance of ligand-based ML models versus physics-based structural modeling methods using the BELKA dataset (three targets: sEH, BRD4, HSA).

• Dataset Stratification: The test set was categorized into three groups based on similarity to the training data:
  • In-Distribution (ID): Same building blocks (BBs) and scaffold (triazine) as training.
  • Near-Distribution (ND): Unseen BBs but same scaffold.
  • Out-of-Distribution (OOD): Unseen BBs and different scaffolds (non-triazine).
• Machine Learning Baselines:
  • Tested various encodings (ECFP4, MACCS, APDP, one-hot SMILES, graphs) and architectures (Random Forests, GCNs, MLPs).
  • Key Strategy: The authors standardized on a Multi-Layer Perceptron (MLP) with specific architectural choices: concatenating ECFP4 and APDP fingerprints, and using shared layers for building blocks 2 and 3 to enforce permutation invariance (accounting for rotational freedom).
  • Data Manipulation: Investigated the impact of downsampling the massive non-hit majority (class imbalance) and random train/test splitting to test generalization.
• Physics-Based Methods:
  • Compared ligand-based ML against three structural methods on the OOD set:
    1. Schrödinger Glide: Standard docking with grid-based scoring.
    2. Rosetta GALigandDock: Co-folding with receptor flexibility.
    3. Boltz-2: A co-folding foundation model predicting binding probabilities directly.
  • Hybrid Approach: Tested whether integrating structural features (Protein-Ligand Interaction Profiler/PLIP fingerprints) derived from docking poses into ML models improved performance.
• Software Tool: Developed DEL-iver, an open-source Python package to streamline DEL data analysis, modeling, and workflow integration.

3. Key Results

A. Limitations of Ligand-Based ML

• Generalization Failure: ML models performed exceptionally well on ID data (e.g., sEH AP = 0.902) but collapsed on OOD data (sEH AP = 0.001, AUROC ~0.5, effectively random).
• Target Dependence: Performance varied significantly by target. BRD4 models showed slightly better OOD generalization than sEH or HSA, but none achieved robust extrapolation to unseen scaffolds.
• Data Composition:
  • Class Imbalance: Surprisingly, removing 99% of non-hit molecules (reducing the dataset from ~100M to ~1M) did not significantly degrade performance, suggesting the dataset size is not the primary bottleneck, but rather data quality and composition.
  • Random Splits: When training on random splits that included OOD molecules (making them "in-distribution" for training), performance on the held-out OOD set improved for BRD4 but degraded for sEH and HSA, indicating that simply increasing diversity does not guarantee better generalization for all targets.

B. Superiority of Structural Methods for OOD

• Hit Discrimination: Structural methods significantly outperformed ligand-only ML models on OOD data.
  • BRD4: Boltz-2 achieved the highest AUROC (0.947) compared to MLP (0.619).
  • sEH: Rosetta GALigandDock achieved the highest AUROC (0.915) compared to MLP (0.501).
• Target Specificity: No single structural method was universally superior. The optimal method depended heavily on the specific protein target and the chemical nature of the ligands.
• Enrichment: Boltz-2 with BRD4 showed an Enrichment Factor (EF) of 86.04 at the top 0.5%, vastly outperforming other methods.

C. Hybrid ML and Structural Features

• Integrating PLIP fingerprints (derived from docking poses) into ML models yielded inconsistent results.
  • For sEH, using GALigandDock-derived PLIPs improved AUROC from 0.265 to 0.572.
  • For BRD4, using Boltz-2-derived PLIPs actually decreased performance despite Boltz-2 having the best standalone docking scores.
• Conclusion: The method with the best hit-discrimination capability does not necessarily generate the most useful features for ML training, likely due to pose inaccuracies or noise in the structural predictions.

4. Key Contributions

1. Benchmarking Generalizability: Provided a rigorous assessment showing that current ML models fail to generalize to OOD chemical spaces in DELs, even with massive datasets.
2. Physics vs. ML: Demonstrated that for OOD hit discrimination, physics-based docking and co-folding (specifically Boltz-2 and GALigandDock) often outperform ligand-based ML, though the choice of method is target-dependent.
3. Data Insights: Revealed that massive dataset sizes are not strictly necessary for high performance (99% of non-hits can be removed) and that data composition (scaffold/BB diversity) is more critical than volume.
4. DEL-iver Package: Released an open-source tool to facilitate the analysis, modeling, and extension of DEL screens, lowering the barrier for computational chemists to reproduce and extend this work.

5. Significance and Conclusion

The paper concludes that rigorous, system-dependent pilot testing is critical before deploying ML models for virtual screening of off-DNA compounds. There is no "one-size-fits-all" solution; the optimal approach (ML vs. Docking vs. Hybrid) depends on the specific protein target and the chemical space being explored.

While ML excels at interpolating within known chemical spaces (ID), it currently cannot reliably extrapolate to novel scaffolds. Structural modeling offers a promising alternative for OOD screening but requires careful selection of the docking protocol. The authors advocate for a workflow where pilot studies define the boundaries of a method's applicability for a given target, rather than assuming generalizability based on aggregated benchmarks. The DEL-iver package serves as a practical resource to implement these best practices.