Empty scaffold allocation bias in molecular distribution shift evaluation

Technical Summary: Empty Scaffold Allocation Bias in Molecular Distribution Shift Evaluation

Problem Definition
Scaffold-based evaluation is a standard methodology for estimating the generalization of molecular machine-learning models to unseen chemical series. This approach relies on the premise that each split unit (a group of molecules sharing a scaffold) corresponds to a chemically meaningful, shared core. However, the Bemis–Murcko scaffold extraction procedure frequently returns an "empty" scaffold representation for molecules lacking a retained ring–linker framework. In standard scaffold splitting protocols, these "no-scaffold" molecules are collapsed into a single, indivisible "empty-scaffold bucket" and treated as a single split unit.

The paper identifies a critical failure mode: this grouping treats an absence-defined residual population as if it were a chemically coherent scaffold class. Because no-scaffold molecules are structurally heterogeneous and lack a common core, allocating them as a single unit can lead to severe allocation bias. Specifically, standard splitting strategies often deplete these molecules from the training set while concentrating them in validation or test sets. This creates an evaluation scenario where model performance is confounded by the underrepresentation of a structurally diffuse, weakly supported subpopulation, rather than testing genuine scaffold-level generalization.

Methodology
The study conducted a systematic analysis across 16 MoleculeNet datasets and 73 Deep-PK/ADMET tasks. The methodology involved:

1. Characterization of No-Scaffold Molecules: The authors quantified the prevalence of empty scaffolds and analyzed their structural cohesion (using Tanimoto similarity and Cohesion Gini) and chemical-space support (nearest-neighbor similarity to scaffold-bearing molecules). They also assessed task-dependent distribution shifts in label and target spaces.
2. Evaluation of Splitting Strategies: Four splitting strategies were compared: standard Bemis–Murcko Scaffold Split, UMAP-based clustering, DataSAIL, and LoHi. The analysis focused on "train-depleted representative folds" (the fold with the lowest no-scaffold proportion in training) to assess worst-case training-side coverage and the resulting enrichment in evaluation branches.
3. Performance Assessment:
   • Classification: Models were evaluated using normalized Precision-Recall (normPR) to measure ranking performance relative to the positive-rate baseline. The study compared performance on no-scaffold test subsets against ordinary-fold references.
   • Regression: Models were evaluated using relative Mean Absolute Error (relMAE) to quantify scale-adjusted errors. The study analyzed "relMAE inflation" (the ratio of error on no-scaffold subsets vs. ordinary folds).
4. Mechanistic Diagnostics: The authors investigated correlations between performance degradation and potential causes, including training-support loss, prior shift, calibration drift, and prediction-scale compression.
5. Proposed Repair (ESA): An "Empty-Scaffold-Aware" (ESA) repair strategy was developed. This post-splitting procedure keeps scaffold-bearing molecule assignments fixed (preserving the leakage-aware backbone) and reassigns no-scaffold molecules into local structural micro-clusters. These micro-clusters are then redistributed across training, validation, and test branches to restore training-side representation while controlling train-facing similarity exposure.

Key Results

• Prevalence and Properties: No-scaffold molecules are common, appearing in 8 of 16 MoleculeNet datasets and 40 of 73 Deep-PK/ADMET tasks, with some physicochemical endpoints approaching 50%. These molecules are structurally diffuse (median within-group Tanimoto similarity of 0.110, close to background levels) and weakly supported by scaffold-bearing chemical space (median nearest-neighbor similarity of 0.300).
• Allocation Bias: Standard Scaffold Split consistently allocates the entire empty-scaffold bucket to evaluation branches, leaving the training set with zero no-scaffold molecules in the worst-case folds. Alternative strategies (UMAP, DataSAIL, LoHi) also fail to guarantee proportional training-side representation, often resulting in training sets with significantly lower no-scaffold coverage than the full dataset.
• Classification Impact: When no-scaffold molecules are enriched in test sets but depleted from training, classification performance degrades significantly. The median normPR drops toward the prior baseline (e.g., from 0.259 to 0.035 in MoleculeNet). This indicates a loss of discriminative ordering, with performance compressing toward random guessing rather than reflecting simple class imbalance.
• Regression Impact: Regression tasks exhibit significant error inflation. The median relMAE increases by 1.35-fold overall, with 44.8% of evaluation points showing at least 2-fold inflation. This inflation is driven by increased absolute prediction error and, in some ADMET tasks, target-scale contraction.
• Mechanistic Drivers: Performance degradation is strongly associated with prediction-scale compression in classification and training-support loss in regression. Prior shift and calibration drift showed weak associations with performance loss.
• ESA Efficacy: The ESA repair strategy successfully restored training-side representation (increasing support recovery from 0.64 to 1.11) and reduced the concentration of no-scaffold molecules in evaluation branches (reducing sink ratio from 5.28 to 0.43) without introducing significant new similarity leakage.

Significance and Claims
The paper claims that empty-scaffold allocation is a previously overlooked failure mode that confounds scaffold-level generalization with the underrepresentation of a weakly supported residual subpopulation. The authors argue that reliable molecular benchmark design must diagnose not only analogue leakage and global train–test similarity but also the representation-induced underrepresentation of such subpopulations.

The study does not claim that scaffold splits are invalid, but rather that their interpretation is compromised when no-scaffold molecules are treated as a single, chemically incoherent unit. The central contribution is the identification of this bias and the demonstration that current splitting strategies do not guarantee adequate training support for these molecules. The proposed ESA repair illustrates a broader principle: evaluation difficulty and training-side representation should be reported as distinct properties of a benchmark split. The authors conclude that benchmark reports should explicitly separate the frequency of no-scaffold molecules, the training-side support they receive, and their enrichment in evaluation branches to distinguish scaffold-level generalization from support-driven extrapolation failure.