On Improving Graph Neural Networks for QSAR by Pre-training on Extended-Connectivity Fingerprints

Technical Summary: On Improving Graph Neural Networks for QSAR by Pre-training on Extended-Connectivity Fingerprints

Problem Statement
Molecular Graph Neural Networks (GNNs) have become prevalent in Quantitative Structure-Activity Relationship (QSAR) studies for drug discovery, yet their superiority over classical molecular featurization methods remains disputed. While GNNs offer the potential to learn abstract, non-linear patterns from molecular graphs, empirical results often show they struggle to outperform classical topological fingerprints, particularly on small, noisy, and heterogeneous experimental datasets common in drug discovery. Furthermore, existing evaluations of GNNs often lack rigorous statistical testing against strong baselines, fail to account for data leakage at the substructural level between pre-training and benchmark datasets, and rely on benchmark splits that may not adequately test out-of-distribution (OOD) generalization.

Methodology
The authors propose a general strategy to improve GNN performance for QSAR by employing self-supervised pre-training on topological fingerprints. Specifically, they utilize Graph Isomorphism Networks (GINs) pre-trained to predict Extended-Connectivity Fingerprints (ECFPs).

• Pre-training Strategy: The GINs are pre-trained on the QMugs dataset (665k molecules) to predict hashed ECFP4 fingerprints (2048-bit vectors). The molecular graphs are featurized using a "substructure tokenization" approach, where local circular substructures (Morgan identifiers) are enumerated and mapped to learnable tokens via a Sort & Slice (S&S) algorithm. This allows the model to learn representations of substructures rather than just atomic types.
• Downstream Evaluation: Pre-trained embeddings are frozen and used as input features for Light Gradient-Boosted Machines (LightGBM) to predict molecular properties. This setup isolates the quality of the molecular representation from the prediction head.
• Benchmarks and Splits: The approach is evaluated on 40 tasks across three datasets: Biogen (6 ADMET regression tasks), MoleculeNet (3 regression, 2 classification), and ChEMBL (binding affinity). To ensure robust OOD evaluation, the authors employ a repeated cross-validation strategy (200 repetitions of 5-fold CV) using Butina clustering on Tanimoto distances to generate evenly sized, challenging OOD splits with minimal molecular overlap.
• Data Leakage Analysis: The study investigates the impact of substructure-level data leakage by filtering the pre-training dataset (QMugs) using varying Tanimoto similarity thresholds (0.5 to 1.0) relative to the benchmark molecules.
• Statistical Rigor: Performance differences are assessed using non-parametric statistical tests (Wilcoxon signed-rank test) with Bonferroni correction and effect size estimation (rank-biserial correlation, $r_{rb}$) across the 200 CV repeats to determine statistical significance.

Key Results

• Performance on Homogeneous Data: On the Biogen dataset, which consists of homogeneous drug-like molecules, ECFP-pre-trained GINs (PT-GIN) achieved statistically significant improvements over all evaluated baselines (including scratch-trained GINs and classical fingerprints) on five out of six regression tasks. The effect sizes were consistently maximal ($r_{rb} = 1.00$), indicating PT-GIN outperformed baselines in every CV repeat for these tasks.
• Performance on Heterogeneous and Complex Data: The benefits of pre-training were not universal. On MoleculeNet tasks with heterogeneous data sources (e.g., Lipophilicity, FreeSolv) and on binding affinity prediction tasks (DRD2, Factor XA), PT-GIN underperformed or showed no significant advantage over classical methods. The authors suggest that data heterogeneity and the need for 3D interaction information (which 2D ECFPs lack) limit the efficacy of this specific pre-training strategy for these endpoints.
• Data Leakage Impact: The analysis of data leakage revealed counterintuitive results. Increasing the similarity between pre-training and benchmark data (i.e., allowing more "leakage") did not consistently improve performance and, in some cases (e.g., Human CLint), led to decreased performance. This suggests that as long as the pre-training corpus contains a diverse vocabulary of chemical substructures, the model learns to generalize to novel substructure combinations, making strict substructural filtering less critical for this specific pre-training objective.
• Virtual Screening: On the challenging MUV virtual screening tasks, models often predicted only negatives, resulting in low Matthews Correlation Coefficients (MCC). No single method demonstrated a clear advantage across all targets, highlighting the difficulty of OOD virtual screening with 2D topological features alone.

Significance and Claims
The paper claims that self-supervised pre-training on topological fingerprints is a simple, robust, and effective strategy for enhancing the OOD performance of GNNs on key QSAR tasks, particularly those involving homogeneous ADMET data. The authors emphasize that this improvement is achieved without increasing model complexity or requiring large, noisy, labeled pre-training datasets.

Crucially, the study highlights that the success of pre-training is task-dependent. While effective for homogeneous regression tasks, it does not universally overcome challenges posed by data heterogeneity or endpoints requiring 3D structural information. The work also contributes a rigorous evaluation framework, demonstrating that statistical testing and effect size estimation are essential for validating performance differences in small-sample QSAR settings. Finally, the findings suggest that for topological pre-training, the diversity of the chemical substructure vocabulary in the pre-training set is more critical than strict exclusion of molecules similar to the benchmark, provided the substructure combinations remain novel.