Do Larger Models Really Win in Drug Discovery? A Benchmark Assessment of Model Scaling in AI-Driven Molecular Property and Activity Prediction

1. Problem Statement

The field of AI-driven drug discovery is currently dominated by a "scale-centric" narrative, suggesting that larger, pretrained foundation models (e.g., Large Language Models, large Graph Neural Networks) will universally supersede compact, task-specific models (e.g., Random Forests, classical QSAR). However, drug discovery datasets are often small, noisy, imbalanced, and chemically clustered. The paper challenges the assumption that increased parameter count or pretraining corpus size automatically translates to superior predictive performance for molecular properties and biological activities. The central question is: Does model scale reliably predict endpoint-level performance, or is performance driven by the alignment between model inductive bias, data regime, and the specific biological endpoint?

2. Methodology

Benchmark Scope and Datasets

The authors evaluated models across 22 distinct endpoints comprising 167,056 held-out task–molecule evaluations:

• Public ADMET (8 tasks): 5 binary classification and 3 regression tasks (e.g., Caco2 permeability, solubility) from the Therapeutics Data Commons.
• Tox21 (12 tasks): 12 binary toxicity assays with high class imbalance (positive rates 2.9–16.2%).
• Anti-Infective (2 tasks):
  • Anti-TB: 49,266 molecules against M. tuberculosis H37Rv (1 µM threshold).
  • Antimalarial: 2,088 molecules against P. falciparum (3D7 and Dd2 strains, 100 nM threshold).

Model Families Compared

The study compared four distinct model families spanning orders of magnitude in complexity:

1. Small ML Models: Classical QSAR learners (Random Forest, ExtraTrees, GBDT, Logistic Regression) trained on engineered features: ECFP4/6 fingerprints, MACCS keys, and RDKit descriptors.
2. Medium Graph Neural Networks (GNNs): Task-specific graph models (GCN, GAT, GIN, Ligandformer) with ~10⁵–10⁶ parameters, learning structural representations directly from molecular graphs.
3. Pretrained Molecular Sequence Models: Transformer-based models (ChemBERTa, ChemBERTa2, MoLFormer) with 10M–77M parameters, fine-tuned from SMILES pretraining.
4. LLM-SAR Reasoning Baselines: Rule-based systems (GPT5.5-SAR, Opus4.7-SAR) where frontier LLMs generate SAR rules (regex/SMARTS) based on training data, which are then applied to held-out folds. These are not fine-tuned predictors but rule-application engines.

Evaluation Protocol

• Validation Strategy: Structure-similarity-separated five-fold cross-validation. Molecules were clustered by ECFP4 fingerprints using MiniBatchKMeans to ensure that structurally similar molecules (chemical neighbors) were not split between training and test sets. This prevents "near-neighbor leakage" and simulates prospective performance better than random splitting.
• Metrics:
  • Classification: PR-AUC (critical for imbalanced Tox21 data) and ROC-AUC.
  • Regression: MAE (Mean Absolute Error) and Pearson Correlation.
• Implementation: All models were optimized with specific hyperparameters (e.g., class weighting for imbalance) to ensure fair comparison within their respective families.

3. Key Results

Overall Performance Winners

Across 167,056 evaluations, larger models did not universally win:

• Classical ML (RF, ExtraTrees): Won 10 primary-metric tasks. They excelled in ADMET regression and many Tox21 assays.
• GNNs (GIN, Ligandformer): Won 9 tasks. They showed strength in specific ADMET endpoints (e.g., DILI, hERG) and the Antimalarial task.
• Pretrained Sequence Models (MoLFormer, ChemBERTa2): Won only 3 tasks (specifically DILI, hERG, and NR-AR/NR-AhR).
• LLM-SAR Baselines: Did not win any primary-metric tasks against the best trained models.

Specific Findings by Domain

• ADMET & Tox21: Classical ML models remained highly competitive or superior, particularly in regression tasks and highly imbalanced toxicity assays. The performance gap between ML and GNNs was often modest and endpoint-dependent.
• Anti-Infective Activity:
  • Anti-TB: RF and ExtraTrees (ECFP4) outperformed MoLFormer and LLM-SAR baselines (PR-AUC ~0.565 vs. ~0.448 for MoLFormer).
  • Antimalarial: GNNs (specifically GIN) led the field (PR-AUC 0.556), outperforming sequence models and LLM-SAR.
• LLM-SAR Performance: While LLM-SAR baselines did not achieve top predictive accuracy, adding "train-fold-derived SAR knowledge" (rules learned from the training set) provided measurable but uneven gains. For example, in Anti-TB, LLM-SAR ROC-AUC improved from ~0.51 to ~0.61 with learned rules, but they still lagged behind supervised predictors.

Regression vs. Classification

• Regression: Classical ML models (especially ExtraTrees with RDKit descriptors) dominated regression tasks (Caco2, Lipophilicity, Solubility), achieving lower MAE and higher Pearson correlation than sequence models.
• Imbalance Sensitivity: For Tox21, PR-AUC was a more discriminative metric than ROC-AUC. Many models appeared successful under ROC-AUC but failed to identify positive classes effectively under PR-AUC, highlighting the danger of relying solely on ROC-AUC in imbalanced drug discovery settings.

4. Key Contributions

1. Empirical Refutation of Universal Scaling: The paper provides rigorous evidence that "bigger is better" is not a universal law in drug discovery. Compact, specialized models often outperform massive foundation models on specific, data-constrained endpoints.
2. Comprehensive Benchmarking: It establishes a large-scale, standardized benchmark covering 22 diverse endpoints (public and internal) with a rigorous structure-separated validation protocol, addressing the issue of data leakage common in previous studies.
3. Inductive Bias Analysis: The study demonstrates that predictive success depends on the alignment between:
   • Molecular Representation: Fingerprints (ECFP) vs. Graphs vs. SMILES tokens.
   • Inductive Bias: Local SAR patterns (favored by RF) vs. Topological patterns (favored by GNNs) vs. Global chemical regularities (favored by Transformers).
   • Data Regime: Small/noisy datasets favor simpler models; large, diverse datasets may benefit from pretraining.
4. Reframing LLMs in Drug Discovery: The authors position LLMs not as direct predictors to replace QSAR models, but as reasoning engines for hypothesis generation, SAR interpretation, and rule extraction. They are valuable for explainability but currently inferior for raw predictive accuracy when labeled data is available.

5. Significance and Implications

• Practical Guidance for Practitioners: Drug discovery teams should not blindly adopt large foundation models. For many standard ADMET and activity prediction tasks, training compact models on fingerprints or descriptors remains the most effective, cost-efficient, and robust approach.
• Validation Protocol Matters: The choice of cross-validation strategy (random vs. structure-separated) drastically alters model rankings. The paper advocates for structure-separated splits to ensure models generalize to new chemical series rather than memorizing local analogues.
• Future Research Direction: The value of large models lies in zero-shot reasoning, mechanistic hypothesis generation, and interpreting complex SAR, rather than replacing supervised predictors for quantitative prediction. Future work should focus on hybrid approaches that leverage the reasoning capabilities of LLMs to guide the feature engineering or architecture of predictive models.

Conclusion: The paper concludes that model scale is a weak explanatory variable for performance in drug discovery. Instead, success is determined by the "Model-Task Fit"—the precise alignment of the model's inductive bias with the specific biological endpoint, data characteristics, and validation protocol.