🤖 machine learning

Multi-Objective Reinforcement Learning for Generating Covalent Inhibitor Candidates

This paper presents a multi-objective reinforcement learning pipeline that successfully generates novel covalent inhibitor candidates for EGFR and acetylcholinesterase, effectively rediscovering known drugs and spontaneously proposing new warhead motifs absent from the training data.

Original authors: Renee Gil

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Renee Gil

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master chef trying to invent a new dish that must satisfy three very difficult requirements at once:

It must taste amazing (bind tightly to the target).
It must be made only from ingredients you can actually buy at the store (it must be synthetically possible).
It must be a "sticky" dish that, once it touches the customer's tongue, locks on forever and won't let go (it must be a covalent inhibitor).

This is the challenge of designing covalent drugs. Unlike normal drugs that just sit next to a disease protein and whisper to it, covalent drugs physically grab onto the protein with a chemical "handshake" that never breaks. This makes them super powerful, but it's also incredibly hard to design them because you have to get the geometry perfect: the "sticky hand" (the warhead) has to be in exactly the right spot to grab the protein.

The Problem: Too Many Choices

Traditionally, scientists would just look through a giant library of existing recipes (molecules) to see if any fit. But this is like looking for a needle in a haystack; you can only find needles that are already in the haystack. You can't invent a new kind of needle.

The Solution: The AI Chef with a GPS

The author of this paper built a Robot Chef (an Artificial Intelligence) that doesn't just look at existing recipes. Instead, it learns the rules of cooking (chemistry) and then tries to invent new dishes from scratch.

Here is how the system works, using a simple analogy:

1. The Apprentice (The Generator)

The AI starts as an apprentice who has read millions of cookbooks (a dataset of known molecules). It knows how to write a recipe (a chemical structure) character by character, like writing a sentence. At first, it just writes random sentences that look like recipes.

2. The Taste Testers (The Scoring Functions)

The AI writes a recipe, and then a panel of Taste Testers (mathematical models) grades it.

The "Can You Make This?" Tester: Checks if the ingredients are too weird or expensive to buy.
The "Sticky Hand" Tester: Checks if the recipe has the right chemical "hand" to grab the protein.
The "Fit" Tester: Simulates putting the dish in the customer's mouth to see if it fits perfectly.
The "Familiarity" Tester: Checks if the dish is too similar to a famous dish (like Osimertinib) or if it's totally new.

3. The Coach (Reinforcement Learning)

This is the magic part. The AI is playing a game where it wants to get the highest score possible.

If the AI writes a recipe that is too hard to make, the Coach says, "Try again, but make it simpler."
If the "Sticky Hand" is missing, the Coach says, "Add a sticky ingredient!"
The AI learns from its mistakes. It tweaks its recipe, tries again, and gets a new score. Over thousands of tries, it learns to balance all these competing demands. It's like a video game character learning to jump, run, and shoot at the same time to get the high score.

4. The "Crowd Control" (Pareto Crowding)

Sometimes, the AI gets too obsessed with one thing (like making the dish super sticky) and forgets the others (like making it easy to cook). To fix this, the Coach uses a "Crowd Control" rule. It forces the AI to keep a diverse menu. It says, "Don't just make 1,000 variations of the same spicy dish. Make some sweet ones, some sour ones, and some that are easy to cook." This ensures the AI explores the whole kitchen, not just one corner.

The Results: What Did the Robot Cook Up?

The researchers tested this AI on two specific targets: EGFR (a protein involved in cancer) and ACHE (a protein involved in memory and nerve function).

Rediscovering Classics: The AI successfully "re-invented" known, successful drugs. It found them about 0.5% to 0.7% of the time. This proves the AI understands the rules of the game.
Perfect Fit: When they took the best recipes the AI made and checked them in a super-computer simulation, the "sticky hands" were positioned perfectly close to the target protein (within 3 to 5 Angstroms—basically, a perfect handshake distance).
The Surprise Ingredients (The Best Part):
The most exciting discovery was that the AI didn't just copy existing recipes. It invented brand new types of "sticky hands" that the scientists had never seen in the training data!
- It created molecules with Allenes (a specific chemical shape).
- It created 3-oxo-β-sultams (a complex ring structure).
- It created α-methylene-β-lactones (a tiny, reactive ring).

None of these specific shapes were in the AI's "cookbooks," but when the scientists looked up scientific literature, they found that these shapes do exist in nature and are known to be great at grabbing proteins. The AI had essentially invented new chemistry that was chemically valid and scientifically sound, even though it had never been taught those specific shapes.

Why This Matters

Think of drug discovery like exploring a dark forest.

Old way: You only walk on the paths that other people have already cleared. You can't find anything new.
This new way: You have a flashlight (the AI) that not only walks the paths but also shines light into the bushes and trees. It finds new trails and new plants that no one knew were there.

This paper shows that we can use AI to not just copy existing drugs, but to imagine new ones that are perfectly tailored to grab onto disease proteins, potentially leading to better cures for cancer and other diseases. The AI is acting as a creative partner for human scientists, suggesting ideas that humans might be too cautious or too focused on tradition to think of.

1. Problem Statement

The rational design of covalent inhibitors is significantly more complex than non-covalent drug design. Covalent inhibitors require a two-step mechanism: initial non-covalent binding followed by an irreversible reaction between an electrophilic "warhead" and a nucleophilic residue on the target protein.

The Challenge: Designers must simultaneously optimize multiple, often competing, objectives: binding affinity, target selectivity, electrophilic reactivity, synthetic accessibility, and the precise geometric positioning of the warhead relative to the target residue.
Limitations of Current Methods: Conventional virtual screening is limited to exploring existing chemical libraries and cannot easily generate novel scaffolds that satisfy these multi-constraint requirements. While Reinforcement Learning (RL) has been applied to general drug discovery, its specific application to the unique constraints of covalent inhibitors remains underexplored.

2. Methodology

The authors developed a Multi-Objective Reinforcement Learning (RL) pipeline tailored for covalent inhibitor generation. The system consists of three core components:

A. Generative Model

Architecture: A SMILES-based Long Short-Term Memory (LSTM) model, implemented via the DrugEx framework.
Pretraining: Instead of training from scratch, the model was initialized using a pre-trained LSTM trained on the Papyrus dataset (1.27 million unique structures from ChEMBL, PubChem, PDBBind, and BindingDB). This ensures the generator produces chemically valid molecules.

B. Scoring Functions (Reward Mechanism)

The RL agent optimizes a weighted sum of several scoring functions. To handle conflicting objectives, the system uses Pareto crowding distance (PCD) to maintain population diversity. Key scorers include:

Viability Constraints (Hard Thresholds):
- Validity: SMILES grammar validity.
- Synthetic Accessibility (SA): Ertl et al. score (threshold $\le$ 6).
- Covalent Activity: Probability of covalent reactivity predicted by a GCNII classifier (threshold $p \ge$ 0.75).
- Residue Affinity: Probability of reacting with specific residue classes (Cys or Ser/Thr) predicted by a custom Graph Attention Network (GAT) classifier (threshold $p \ge$ 0.75).
- Docking Score Approximation: A Graph Neural Network (GNN) approximating AutoDock Vina scores (threshold $\le$ -6.0 kcal/mol).
Optimization Objectives:
- Structural Overlap: 3D alignment score against known reference inhibitors (e.g., Osimertinib for EGFR).
- Tanimoto Similarity: Fingerprint similarity to reference scaffolds (e.g., Quinoline).
- QED: Quantitative Estimate of Drug-likeness.

C. Reinforcement Learning Algorithm

Optimization: Uses a policy gradient scheme (following Šícho et al.) to maximize the reward $R^*$ .
Diversity Maintenance: Solutions are ranked by Pareto dominance. Within each front, Pareto Crowding Distance (PCD) is calculated to favor solutions that are well-spread in the objective space, preventing the generator from collapsing into a single local optimum.

D. Evaluation & Warhead Tagging

Warhead Identification: Since manually identifying warheads in generated structures is impractical, the authors used GradCAM (Gradient-weighted Class Activation Mapping) adapted for GNNs. Atoms with normalized activation $>0.3$ were tagged as the warhead region.
Validation: Top candidates underwent full docking (using DOCKSTRING) to verify warhead-to-residue distances (e.g., distance to Cys797 for EGFR, Ser200 for ACHE).

3. Key Contributions

Specialized RL Pipeline: The first application of multi-objective RL specifically tailored for covalent inhibitor generation, incorporating residue-specific affinity classifiers and warhead geometry constraints.
Novel Scoring Models: Development of a GAT-based residue affinity classifier and a GNN-based docking score approximator trained on the ProteinReactiveDB dataset.
Discovery of Atypical Warheads: The pipeline spontaneously generated chemical scaffolds containing warhead motifs absent from the training data (ProteinReactiveDB), including:
- Allenes
- 3-Oxo-β-sultams
- $\alpha$ -Methylene- $\beta$ -lactones
- Significance: These structures were validated against literature as known covalent warheads, demonstrating the model's ability to extrapolate beyond its training distribution.

4. Results

The pipeline was tested on two targets: EGFR (targeting Cys797) and Acetylcholinesterase (ACHE) (targeting Ser200).

Rediscovery Rates:
- EGFR: Up to 0.50% rediscovery rate (EGFR-3 model) in 10,000 generated structures.
- ACHE: Up to 0.74% rediscovery rate (ACHE-3 model).
Geometric Validation:
- EGFR: Best candidate achieved a warhead-to-Cys797 distance of 5.5 Å (comparable to the 3.35 Å in the Osimertinib crystal structure).
- ACHE: Best candidate achieved a warhead-to-Ser200 distance of 3.2 Å (carbonyl path).
Chemical Space Exploration:
- The models successfully rediscovered known inhibitors (e.g., Osimertinib, Malathion derivatives).
- Crucially, the models generated atypical warheads (e.g., $\alpha$ -methylene- $\beta$ -lactones) not seen in the training set but supported by independent literature, proving the system can explore novel regions of covalent chemical space.
Model Performance:
- The docking approximation models achieved high accuracy ( $R^2 = 0.86$ for EGFR, $0.75$ for ACHE).
- Adding too many constraints (e.g., in EGFR-4 and ACHE-4) reduced the number of "desirable" structures, highlighting the difficulty of satisfying all objectives simultaneously.

5. Significance and Conclusion

This work demonstrates that Multi-Objective RL is a powerful tool for covalent drug discovery.

Beyond Training Data: The most significant finding is the model's ability to "hallucinate" chemically valid, novel warhead scaffolds that were not present in the training data but are chemically sound. This suggests RL can effectively navigate chemical space beyond the distribution of known compounds.
Practical Utility: The pipeline serves as a viable starting point for medicinal chemists, generating candidates with correct warhead-residue geometries and high synthetic accessibility.
Future Outlook: While the study relies on approximations (GNN docking scores, GradCAM warhead tagging), the results suggest that combining generative AI with specific covalent constraints can accelerate the discovery of next-generation covalent therapeutics, particularly for targets with shallow pockets or resistance mutations.

Limitations: The authors acknowledge that the scoring functions are themselves ML models trained on limited data, potentially inheriting biases. Furthermore, the warhead distances are estimates; experimental validation remains the definitive test for inhibitory activity.