Source- and Release-Aware Calibration Recovers EGFR Potency Prediction Under Cross-Source and Future-Release Domain Shift

Technical Summary: Source- and Release-Aware Calibration for EGFR Potency Prediction

Problem Statement
Ligand-based potency models for the Epidermal Growth Factor Receptor (EGFR) often demonstrate strong internal accuracy when evaluated on random splits. However, the paper argues that these metrics are inflated by shared analog series, duplicate chemistry, and source-specific assay conventions. In real-world medicinal chemistry deployment, models face severe domain shifts across:

1. Scaffold and Temporal shifts: Generalizing to new chemical scaffolds or future time periods.
2. Cross-source shifts: Transferring from ChEMBL to independent databases like BindingDB.
3. Future-release shifts: Predicting on molecules added to a database after the training cutoff (e.g., ChEMBL36 molecules when trained on ChEMBL30).

Under strict evaluation protocols, the paper demonstrates that standard models (fingerprints, ensembles, graph neural networks, and even pairwise activity-cliff-aware models) collapse when evaluated on these shifted distributions. Specifically, on a true future-release panel of 2,907 non-overlapping ChEMBL36 molecules, ChEMBL30-trained models achieve negative $R^2$ scores, performing worse than a simple mean baseline.

Methodology
The study employs a rigorous, reproducible framework to diagnose and recover from these failures:

• Strict Data Curation: A high-quality EGFR IC50 benchmark was reconstructed from ChEMBL (and BindingDB) using strict rules: exact equality relations, nM units, valid assay types, removal of duplicate conflicts (SD > 0.5), and scaffold-based splitting.
• Benchmark Construction:
  • Future-Release Panel: Training was locked to ChEMBL30, and evaluation was performed on 2,907 non-overlapping molecules from ChEMBL36.
  • Cross-Source Panel: Evaluation on BindingDB data with strict overlap removal.
  • Public Benchmarks: Evaluation on MoleculeACE and ACNet activity-cliff datasets.
• Baseline Models: The study tested a wide range of architectures, including Morgan fingerprints, Ridge/RandomForest/XGBoost, and a modern Directed Message Passing Neural Network (GINE/D-MPNN).
• Proposed Solution: Source- and Release-Aware Residual Calibration:
  • Instead of retraining the base model, the authors propose a lightweight, backpropagation-free calibration head.
  • Mechanism: A small set of "support" labels (tens to hundreds of molecules) from the shifted distribution (e.g., BindingDB or ChEMBL36) is used to estimate the residual error $r(x) = y - f(x)$ of the base model.
  • Calibration Heads: Several estimators were tested, including a mean-offset, linear regression, Morgan residual ridge (regressing residuals on fingerprints), and Morgan kNN-residual (transferring residuals from nearest neighbors).
  • Prediction: The calibrated prediction is $\hat{y}_{cal}(x) = f(x) + \hat{r}(x)$.
  • Uncertainty: Split conformal prediction intervals were recalibrated using residuals from the shifted support set to restore valid coverage.

Key Results

1. Model Collapse Under Shift:

   • On the ChEMBL30 $\to$ ChEMBL36 future-release panel, uncalibrated models (including GNNs) achieved $R^2$ values ranging from -0.33 to -2.52, failing to beat the mean baseline.
   • Similar collapses were observed on BindingDB and temporal external splits.
   • Pairwise activity-cliff objectives improved local SAR metrics but did not solve the global domain shift.

2. Recovery via Few-Shot Calibration:

   • Using only a small number of scaffold-disjoint support labels from the shifted distribution, the residual calibration head restored positive $R^2$.
   • BindingDB: RMSE improved from 1.5959 (zero-shot) to 0.8285 with 512 support labels.
   • Future-Release (ChEMBL36): RMSE improved from 1.42 ($R^2$ -0.33) to 1.00 ($R^2$ +0.33) with just 64 labels, and to 0.85 ($R^2$ +0.52) with 512 labels.
   • The method was effective with as few as 16–64 distribution-matched labels.

3. Generalization and Efficiency:

   • Target Transfer: The same calibration recipe successfully recovered performance on independent non-EGFR kinase targets (ABL1 and BRAF) where the base model had zero-shot $R^2$ values of -1.81 and -2.45, respectively.
   • Compute Efficiency: At small label budgets (16–64 labels), the backprop-free calibration matched or outperformed full GNN fine-tuning while requiring roughly an order of magnitude less compute time (e.g., 0.01s vs 0.38s).
   • Uncertainty: In-domain conformal intervals severely under-covered future-release data. Recalibrating the conformal radius on a few shifted labels restored near-nominal coverage (e.g., 0.90 target $\to$ 0.90 empirical coverage), albeit with wider, more honest intervals.

Significance and Claims
The paper claims that the observed degradation in EGFR potency prediction is not an inherent failure of representation learning (as GNNs also failed) but a solvable domain shift problem. The central contribution is a practical, actionable deployment recipe:

• Do not rely solely on representation transfer or internal validation metrics.
• Deploy models for in-domain triage but collect a small, distribution-matched support set (tens of molecules) when moving to cross-source or future-release scenarios.
• Apply residual calibration to this support set to recover valid predictions and uncertainty estimates.

The authors emphasize that this approach turns a "failing" release-forward model into a usable one without the computational cost or risk of catastrophic forgetting associated with full fine-tuning. The study also highlights that activity-cliff-aware objectives are valuable for local SAR but insufficient for global domain adaptation without source-aware calibration.

Limitations Acknowledged
The paper notes that strict curation reduces activity-cliff counts, that few-shot calibration assumes the availability of target-source labels, and that while the ChEMBL30-to-36 experiment is a strong retrospective stress test, true prospective wet-lab validation remains the highest level of evidence. Additionally, the current XAI labels are weak structural annotations rather than expert-curated pharmacophores.