HEDGEHOG: Hierarchical Evaluation of Drug Generators Through Rigorous Filtration

Technical Summary: HEDGEHOG Benchmark for Molecular Generators

Problem Statement
Generative molecular models are increasingly used to propose new candidate compounds de novo for early drug discovery. However, current evaluation practices rely heavily on intrinsic metrics—such as validity, novelty, uniqueness, and distributional similarity—which fail to capture whether generated molecules are medicinally plausible or suitable for downstream computational workflows. Standard benchmarks often evaluate objectives like docking scores or synthetic accessibility in isolation or simplified combinations. This approach can produce false positives, leading to incorrect assumptions about model utility and inefficient use of computational resources. The core problem is the gap between "chemically valid" generation and "practically useful" generation, where molecules may satisfy isolated criteria but fail simultaneously under the multiparameter constraints of medicinal chemistry, synthesis, and 3D binding pose plausibility.

Methodology: The HEDGEHOG Protocol
To address these limitations, the authors introduce HEDGEHOG (Hierarchical Evaluation of Drug Generators Through Rigorous Filtration), a unified six-stage filtration benchmark inspired by industrial hit identification workflows. The protocol employs a coarse-to-fine cascade, removing implausible molecules early to conserve computational resources for expensive downstream stages. The six stages are:

1. Molecular Preprocessing: Standardization of SMILES strings, removal of salts/solvents, disconnection of metals, and filtering of invalid valence states or unsupported atom types using RDKit and DataMol.
2. Physicochemical Descriptor Screening: Computation of 21 descriptors (e.g., lipophilicity, polarity, ring statistics, sp3-hybridized carbon fraction) against configurable thresholds to remove extreme outliers.
3. Structural Filters: Removal of molecules containing undesirable substructures or poor medicinal chemistry profiles. This stage utilizes approximately 2,460 SMARTS patterns and 10 rule sets (including PAINS, BMS, Glaxo, and Lilly Medchem Rules) alongside graph-level sanity checks.
4. Synthesis Feasibility: A dual approach combining heuristic scores (SA, RA, SYBA) with explicit retrosynthetic route finding using AiZynthFinder. This stage tests whether at least one synthetic route exists under a specific setup.
5. Docking and Binding Affinity Estimation: Surviving molecules are docked into the KRAS G12D switch-II pocket (PDB: 7EW9) using three engines: smina, GNINA, and Matcha. Binding affinity is estimated using Boltz-2. This stage requires agreement across multiple models to avoid tool-specific false positives.
6. Three-Dimensional (3D) Filtration: Post-docking checks for pose quality, conformer deviation, pocket containment, and target-specific interactions (specifically a required interaction with Asp12 for the KRAS G12D instance).

The benchmark was applied to 23 molecular generators across three classes: unconditional, ligand-based, and protein-based models. Each model generated 10,000 distinct SMILES strings, resulting in a total evaluation set of 230,000 molecules.

Key Results
The study reveals a drastic attrition rate across the filtration pipeline:

• Overall Survival: Only 0.65% of the initial 230,000 molecules (1,490 total) survived all six stages.
• Stage-wise Attrition:
  • Preprocessing: Conditional models (ligand/protein-based) showed higher pass rates (97%) compared to unconditional models (75%).
  • Physicochemical & Structural Filters: These stages acted as major bottlenecks. After structural filtering, only ~5% of molecules remained. This indicates that many molecules satisfying physicochemical descriptors still contain prohibited substructures.
  • Synthesis Feasibility: Explicit retrosynthetic search was more selective than heuristic scores alone. Conditional pass rates varied significantly; for example, ligand-based models retained 35.89% of molecules passing structural filters, while unconditional models retained 59.72%.
  • Docking & 3D: Unconditional models retained the highest fraction of molecules relative to their initial set at the docking stage (1.80%), whereas protein-based models had the lowest (0.96%). However, protein-based models showed the highest conditional survival at the final 3D stage (63.15% of docking survivors), suggesting their retained poses are geometrically more plausible once they pass earlier filters.
• Top Performers:
  • Dragonfly (protein-based) was the top performer, yielding 345 final survivors. Its success is attributed to combining interactome-based learning with molecular generative priors and the ability to incorporate ligand templates and 3D binding site information.
  • REINVENT4 (V) (ligand-based) and REINVENT4 (unconditional) followed with 182 and 163 survivors, respectively.
  • The results highlight that strong performance on intrinsic metrics does not guarantee survival in a realistic triage workflow. For instance, some models with high early-stage retention failed to produce viable candidates due to structural liabilities or synthesis intractability.

Significance and Claims
The paper claims that HEDGEHOG serves as a "stress test" for molecular generators, exposing failure modes invisible to conventional intrinsic metrics. Its primary contributions are:

1. Reframing Evaluation: It shifts the evaluation paradigm from unconstrained molecule generation to the proposal of "useful chemical space" that satisfies a sequence of medicinal-chemistry-oriented constraints.
2. Diagnostic Value: By reporting conditional survival at each stage, HEDGEHOG identifies specific failure points (e.g., structural alerts vs. synthesis feasibility) and separates architectural strengths from training objective limitations.
3. Realistic Benchmarking: The benchmark demonstrates that molecules appearing acceptable under isolated criteria rarely satisfy medicinal chemistry, synthesis, docking, and 3D pose filters simultaneously.
4. Practical Utility: It provides a more informative basis for developing generative models aligned with early drug discovery, emphasizing that the inability to maintain chemical realism, synthesizability, and target compatibility simultaneously is the central limitation of current models.

The authors note that while the benchmark is stringent and highly discriminative, it is currently instantiated for the KRAS G12D target. They acknowledge limitations, such as the lack of variance estimation across generation seeds for all models and the absence of long-term ADMET or clinical success predictors, suggesting these as areas for future extension.