🤖 AI

PGFS++: Molecular Property Improvement under Synthesis and Diversity Constraints

This paper introduces PGFS++, a synthesis-aware reinforcement learning framework that improves molecular properties while preserving output diversity and providing explicit synthesis routes by addressing the reactant selection limitations and reward-hacking diversity collapse found in its predecessor, PGFS+.

Original authors: Boqiao Zhang, Godbless James, Sai Krishna Gottipati, Andrew Fitzgibbon

Published 2026-08-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Boqiao Zhang, Godbless James, Sai Krishna Gottipati, Andrew Fitzgibbon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the early stages of creating new medicines, scientists face a constant balancing act. They need to design molecules that are potent enough to fight disease, but they also need to ensure those molecules can actually be built in a laboratory. For years, artificial intelligence has helped generate new molecular structures, often optimizing them for specific traits like how well they might bind to a virus or how safe they are for the human body. However, a significant gap has remained: many of these computer-generated designs are chemically valid on paper but impossible to manufacture with the tools and ingredients available in real-world labs. To bridge this gap, researchers have turned to a method called forward synthesis, which treats drug creation like a step-by-step construction project, starting with a known molecule and adding pieces one by one using established chemical reactions. The goal is to guide the computer not just to a better molecule, but to a better molecule that comes with a clear, feasible instruction manual for how to build it.

A team of researchers at the University of Cambridge and Graphcore has developed a new system called PGFS++ to solve a specific problem that arose when trying to automate this process. Their work builds on an earlier method known as PGFS, which used a type of artificial intelligence learning called reinforcement learning. In this setup, the computer acts as an agent that tries to improve a starting molecule by selecting chemical reactions and building blocks. The previous version of this system, an upgrade they called PGFS+, worked well at finding molecules with improved properties, but it suffered from a strange flaw. The system became too efficient at finding a single, highly rewarding solution. Instead of creating a wide variety of improved versions for different starting materials, it would funnel hundreds of different input molecules into the exact same final product. The researchers call this the "magnet" effect, where the system is pulled toward one high-scoring outcome, collapsing the diversity of results and making the tool useless for creating tailored improvements for specific drugs.

To fix this, the team introduced PGFS++, a refined framework that forces the computer to keep the connection between the starting material and the final result. The system still works by selecting reaction templates and compatible building blocks from a massive library of over 118,000 available chemical ingredients. It uses a pre-computed map to quickly identify which building blocks can legally react with a specific part of the molecule, narrowing down millions of possibilities to a manageable list. The key innovation lies in how the system rewards itself. In the previous version, the computer was only rewarded if the final molecule had better properties, which encouraged it to ignore the starting point entirely. In PGFS++, the reward system includes a bonus for maintaining structural similarity to the original input. This means the computer is penalized if it strays too far from the starting molecule, ensuring that the improvements are specific to that particular drug candidate rather than a generic, one-size-fits-all solution.

The results of this new approach were tested on two common measures of drug quality: a score for how "drug-like" a molecule is and a score for how easy it is to synthesize. When the researchers ran simulations with thousands of different starting molecules, the new system successfully improved the target properties while keeping the output diverse. Unlike the earlier version, which produced nearly identical molecules for almost every input, PGFS++ generated a wide range of unique, improved structures. The system also maintained high scores for synthetic accessibility, meaning the routes it found were practical for chemists to follow. By preventing the collapse into a single "magnet" solution, the researchers demonstrated that it is possible to optimize drug candidates for specific needs without sacrificing the variety required to explore different chemical possibilities. This work suggests that future drug discovery tools can be both highly effective at improving molecular properties and respectful of the unique starting points that define real-world research.

Technical Summary: PGFS++ for Molecular Property Improvement

Problem Statement

In early-stage drug discovery, improving molecular properties (e.g., drug-likeness or binding affinity) is critical. However, molecules optimized in an unconstrained chemical space often lack practical value if they cannot be synthesized. While Reinforcement Learning (RL) methods like PGFS (Policy Gradient for Forward Synthesis) address this by constraining search to synthetically accessible routes, they suffer from specific limitations:

Indirect Reactant Selection: PGFS uses a neural network to predict a continuous embedding for the second reactant ( $R_2$ ), followed by a nearest-neighbor search to retrieve a discrete molecule. This creates a mismatch between the optimized continuous action and the executed discrete reaction, potentially limiting learning effectiveness.
The "Magnet Output" Failure Mode: Even improved variants (like the proposed PGFS+) can collapse output diversity. The policy learns to map diverse input molecules to a single high-reward "magnet molecule." While this maximizes the target property score (e.g., QED), it destroys output diversity and fails to achieve input-specific molecular improvement.

Methodology

The paper proposes PGFS++, a synthesis-aware RL framework designed to improve molecular properties while preserving diversity and ensuring explicit synthesis routes.

1. Enhanced Action Space and Scoring (PGFS+)

To address the limitations of PGFS, the authors first developed PGFS+:

Learned Embeddings: Instead of predicting continuous embeddings, PGFS+ uses trainable embedding lookup tables for reaction templates and second reactants ( $R_2$ ).
Direct Global Scoring: Molecules and templates are projected into a learned reaction space. A query vector $q = \text{concat}(\phi_1(R_1), \phi_T(T))$ is used to score all template-compatible $R_2$ candidates directly via an attention-like mechanism.
Algorithm Shift: The continuous-action algorithm (TD3) is replaced with PPO (Proximal Policy Optimization), which naturally supports a categorical policy over the indexed $R_2$ action space.
Pre-computed Masks: To handle the large action space, the authors pre-compute compatibility masks using RDKit substructure matching, reducing the candidate set to a tractable size (median ~1,000 candidates per template).

2. Mitigating Diversity Collapse (PGFS++)

To resolve the "magnet output" problem where diverse inputs map to identical high-reward outputs, PGFS++ introduces an Input-Output Similarity Bonus into the reward function.

Reward Shaping: The episode reward $r_{episode}$ is modified to include a term based on the Tanimoto similarity between the input ( $m_{in}$ ) and output ( $m_{out}$ ) molecules:
$r_{episode} = r_{property}(m_{out}) \cdot (1 + c \cdot \text{sim}(m_{in}, m_{out}))$
Reward Capping: If the similarity falls below a threshold $\tau$ , the reward is capped to prevent the policy from ignoring the input structure entirely.
Theoretical Justification: The authors derive that without this constraint, the policy induces an "inward edit bias" (negative covariance between input and edit displacement), leading to diversity collapse ( $D_{out} \to 0$ ). The similarity bonus tightens the upper bound on diversity drop, encouraging the model to produce input-specific improvements rather than converging to a single magnet molecule.

Key Contributions

PGFS++ Framework: A synthesis-constrained RL framework that extends PGFS by utilizing discrete action spaces, pre-computed compatibility masks, and global attention-based scoring for reactant selection.
Identification of Magnet Output: The paper identifies and characterizes a specific failure mode in synthesis-aware molecular improvement where high property scores are achieved at the cost of output diversity.
Input-Output Similarity Bonus: The proposal of a multiplicative similarity bonus as an effective mechanism to resolve diversity collapse, ensuring that the model performs input-specific improvements while maintaining high output diversity.

Experimental Results

The framework was evaluated on a dataset of ~118,000 molecules and 102 reaction templates, focusing on improving QED (Quantitative Estimate of Drug-likeness) and SEH (Synthetic Accessibility/Health).

Performance vs. Baselines: PGFS++ consistently outperformed PGFS and its variants (PGFS-TAC, PGFS-MB) in both property improvement and diversity metrics. It achieved performance competitive with REINVENT4 (a generative model) and SynFlowNet (a GFlowNet-based method), with the distinct advantage of providing explicit synthesis routes.
Diversity Preservation: Unlike PGFS+, which collapsed diversity (producing the same output for ~97.8% of test inputs), PGFS++ maintained high output diversity while achieving high mean QED.
Ablation Studies: Comparisons between additive and multiplicative similarity bonuses showed that the multiplicative form (used in PGFS++) better preserves the target property (QED) while effectively mitigating the magnet output problem.
Metrics: The framework demonstrated superior performance in Route-Aware Synthesizability (RAS), input-output similarity, and structural diversity compared to baselines that lack synthesis constraints or suffer from diversity collapse.

Significance and Claims

The paper claims that PGFS++ represents a significant step forward in input-specific molecular improvement. By treating the input molecule as the start of a forward-synthesis trajectory and enforcing structural similarity, the framework avoids the "reward hacking" behavior where models optimize for a global maximum at the expense of diversity.

The authors emphasize that while previous methods either ignored synthesis constraints or failed to maintain diversity under those constraints, PGFS++ successfully balances three competing objectives:

Property Improvement: Maximizing target metrics like QED.
Synthetic Feasibility: Providing explicit, valid synthesis routes.
Diversity: Ensuring the output molecules are structurally diverse and specific to their inputs, rather than collapsing to a single "magnet" solution.

The work suggests that for practical drug discovery, where maintaining a scaffold or specific structural features of a lead compound is often required, synthesis-aware methods must explicitly penalize the loss of input-output relatedness to be effective.