Do Larger Models Really Win in Drug Discovery?A Benchmark Assessment of Model Scaling in AI-Driven Molecular Property and Activity Prediction

1. Problem Statement

The field of drug discovery is currently dominated by a "scale-centric" paradigm, driven by the success of large language models (LLMs) and molecular foundation models. There is a prevailing assumption that larger, general-purpose pretrained models will inevitably supersede traditional, compact cheminformatics models and task-specific Graph Neural Networks (GNNs) in predictive tasks.

However, this assumption lacks rigorous empirical validation across diverse molecular endpoints. The core problem addressed by this study is whether model size and generalization capability actually translate to superior predictive performance in specific drug discovery contexts (ADMET, toxicity, and anti-infective activity), or if specialized, smaller models remain more effective.

2. Methodology

The authors conducted a comprehensive benchmark assessment designed to eliminate data leakage and ensure fair comparison across different model architectures.

• Datasets & Scope:
  • Endpoints: 22 distinct molecular property and activity endpoints.
  • Sources: Public benchmarks (ADMET, Tox21) and two internal anti-infective activity datasets (anti-TB and antimalaria).
  • Evaluation Scale: A total of 167,056 held-out task–molecule evaluations.
• Validation Protocol:
  • Structure-Similarity-Separated Five-Fold Cross-Validation: This is a critical methodological choice. Unlike random splits, this approach ensures that molecules in the test set are structurally dissimilar to those in the training set, rigorously testing a model's ability to generalize to new chemical space rather than just memorizing similar structures.
• Models Evaluated:
  • Classical Machine Learning (ML): Random Forest (RF) using ECFP4 fingerprints and ExtraTrees using RDKit descriptors.
  • Graph Neural Networks (GNNs): GIN (Graph Isomorphism Network) and Ligandformer.
  • Pretrained Molecular Sequence Models: MoLFormer and ChemBERTa2 (representing the "large model" category).
  • Rule-Based SAR Reasoning Baselines: GPT5.5-SAR and Opus4.7-SAR (representing LLM-based reasoning).
• Metrics: Performance was measured using primary metrics specific to each endpoint, with a focus on predictive accuracy and generalization.

3. Key Contributions

• Systematic Benchmarking: The study provides one of the largest and most rigorous comparative analyses of model scaling in drug discovery, covering over 160k evaluations across diverse chemical spaces.
• Challenge to the "Scale Hypothesis": It empirically tests the industry trend that "bigger is better," specifically isolating the impact of model size and pretraining against specialized architectures.
• Differentiation of Use Cases: The paper distinguishes between predictive performance (quantitative accuracy) and interpretive/reasoning capabilities (qualitative insights), offering a nuanced view of where large models actually add value.

4. Key Results

The results challenge the notion that larger models universally dominate:

• Winning Distribution:
  • Classical ML (RF, ExtraTrees): Won 10 of the primary-metric tasks.
  • GNNs (GIN, Ligandformer): Won 9 tasks.
  • Pretrained Sequence Models (MoLFormer, ChemBERTa2): Won only 3 tasks.
  • Rule-Based SAR (GPT5.5, Opus4.7): Did not win any tasks under the prespecified primary metrics.
• Performance Characteristics:
  • The performance differences between classical ML, GNNs, and pretrained sequence models were often modest and highly endpoint-dependent.
  • Larger or more general models did not provide a universal predictive advantage.
  • SAR Reasoning: While rule-based SAR baselines did not win on primary prediction metrics, incorporating SAR knowledge derived from training folds provided measurable (though uneven) gains in reasoning and interpretation tasks.

5. Significance and Implications

• Re-evaluating Model Selection: The findings suggest that for the specific task of molecular property and activity prediction, compact, specialized models remain highly effective. Drug discovery practitioners should not automatically default to massive foundation models without considering the specific data regime and endpoint biology.
• Context-Dependent Performance: Predictive success is not solely a function of model size. It depends on the alignment between:
  • Molecular representation (e.g., fingerprints vs. graphs vs. sequences).
  • Inductive bias of the model.
  • The data regime (size and diversity).
  • The specific biology of the endpoint.
  • The validation protocol (e.g., structure-separated splits).
• Role of Large Models: While large models may not currently outperform specialized models in raw prediction accuracy, they retain potential value in zero-shot reasoning, Structure-Activity Relationship (SAR) interpretation, and hypothesis generation.
• Future Direction: The paper advocates for a shift from a "scale-centric" view to a "fit-for-purpose" approach, where model architecture is chosen based on the specific constraints and goals of the drug discovery task rather than model parameter count alone.