🤖 machine learning

BERTology of Molecular Property Prediction

This paper systematically investigates the inconsistent performance of Chemical Language Models (CLMs) in molecular property prediction by conducting hundreds of controlled experiments to analyze the effects of dataset size, model size, and standardization, aiming to provide comprehensive numerical evidence and clarify the underlying mechanisms that are often overlooked in existing literature.

Original authors: Mohammad Mostafanejad, Paul Saxe, T. Daniel Crawford

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mohammad Mostafanejad, Paul Saxe, T. Daniel Crawford

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the "language" of chemistry. In this world, molecules aren't just atoms floating around; they are written as strings of text, like secret codes (called SMILES). For example, a molecule might look like a sentence: CC(=O)OC1=CC=CC=C1.

For years, scientists have tried to teach robots to predict how these molecules will behave (like whether a drug will cure a disease or if it's toxic) using standard math tricks. But recently, a new, powerful tool arrived: Chemical Language Models (CLMs), which are based on a famous AI architecture called BERT. These models are like super-smart students who read millions of chemical "sentences" to learn the grammar and vocabulary of chemistry before being tested on specific tasks.

However, there's a problem. Some scientists say these models are amazing, while others say they are failing or even getting worse when you give them more data. It's like a debate where half the class says, "More study time makes you smarter!" and the other half says, "No, studying too much makes you confused!"

This paper is the authors' attempt to settle the debate by running hundreds of carefully controlled experiments. They act like detectives trying to figure out exactly why these models behave the way they do.

Here is the breakdown of their findings using simple analogies:

1. The "Standardization" Problem: The Dictionary Mix-Up

Imagine you are teaching a child to read. You give them a book written in perfect, standard English. But then, you start mixing in pages from a book written in a different dialect, where the same word is spelled differently or means something slightly different. The child gets confused.

In chemistry, different databases (like PubChem and ChEMBL) have different rules for writing the same molecule. One might write a molecule with a specific charge, while the other neutralizes it.

The Finding: The authors discovered that if you mix these different "dialects" (standardization protocols) in the training data, the model gets confused and performs poorly. It's like trying to learn French while someone keeps switching to Spanish mid-sentence.
The Good News: Bigger, smarter models (like the "Base" version) are more resilient. They can handle a little bit of confusion better than the smaller, "Tiny" models. But the best strategy is to keep the "language" consistent.

2. The "Scaling" Law: Bigger is Better (Usually)

There is a famous rule in AI called "Scaling Laws," which basically says: The more data you feed a model, and the bigger the model is, the smarter it gets.

The Finding: The authors confirmed this holds true for chemical models too.
- More Data: If you feed the model more chemical sentences, it learns better.
- Bigger Models: A "Base" model (with more brainpower) learns faster and more efficiently than a "Tiny" model.
- The Catch: There is a point of diminishing returns. If you have a tiny model, giving it a massive amount of data doesn't help much because the model is too small to understand it all. But if you have a big model, it can absorb that data and get significantly smarter.

3. The "Randomness" Myth: It's Not About Luck

Sometimes, when you run an experiment, the results change just because you started with a different random number (like rolling a die to decide where to start).

The Finding: The authors found that while randomness (like the starting weights of the model or which data you pick first) does cause small changes, it is not the main reason for the big differences in performance. The size of the model and the quality of the data matter way more than the luck of the draw.

4. The "Tokenization" Question: How to Cut the Words

Before a computer reads a sentence, it has to chop it up into smaller pieces called "tokens" (like cutting a sentence into words or syllables). There are different ways to do this (WordPiece vs. BPE).

The Finding: It turns out it doesn't matter too much which cutting method you use. As long as you have a good model and good data, both methods work almost the same. The "cutting style" isn't the secret sauce.

5. The Real-World Test: Does it Actually Work?

Finally, they tested these models on real-world drug discovery tasks (predicting things like how well a drug dissolves in the body or how fast the liver breaks it down).

The Result: The big, well-trained BERT models performed just as well as, or sometimes better than, the traditional, older machine learning methods.
The Trade-off: Training these big models is expensive and takes a long time (like building a supercomputer). Traditional methods are cheap and fast (like using a calculator).
- The Verdict: If you have the resources, the big model is worth it because it understands the "language" of chemistry deeply and can be adapted to many different tasks. If you are in a rush or have a tiny budget, the old methods are still fine.

Summary: The Takeaway

Think of this paper as a guidebook for building a chemistry robot:

Don't mix your languages: Make sure all your chemical data is written in the same "dialect" (standardized) before you teach the robot.
Size matters: If you want the best results, use a bigger model and feed it as much high-quality data as you can.
Don't worry about the dice: Random luck isn't the main factor; focus on the data and the model size.
It works: These AI models are powerful tools for drug discovery, but they require a significant investment of time and money to train properly.

The authors have essentially provided the "instruction manual" for how to stop the confusion in the field and start building reliable, high-performance chemical AI.

1. Problem Statement

Molecular Property Prediction (MPP) is critical for drug discovery and materials design. While Chemical Language Models (CLMs) based on Transformers (specifically BERT architectures) have emerged as strong competitors to classical machine learning (ML) models, recent literature reports inconsistent and contradictory results regarding their performance.

The Paradox: Some studies suggest that increasing pre-training dataset size or model size improves performance, while others report performance degradation or stagnation.
The Gap: There is no established "scaling law" framework for encoder-only models with Masked Language Modeling (MLM) objectives in the chemical domain, unlike the well-defined scaling laws for auto-regressive Large Language Models (LLMs).
The Confounders: It is unclear whether inconsistencies arise from dataset size, model size, tokenization methods, or standardization noise (mixing SMILES strings processed by different chemical databases with different standardization protocols).

2. Methodology

The authors conducted a systematic investigation involving hundreds of meticulously controlled experiments to isolate the effects of specific variables on BERT's pre-training and fine-tuning performance.

Experimental Setup

Pre-training Data: The PubChem dataset (approx. 119 million unique compounds, standardized canonical isomeric SMILES).
Fine-tuning Data: Biogen's ADME dataset (Absorption, Distribution, Metabolism, Excretion) covering six regression endpoints (HLM, RLM, hPPB, rPPB, MDR1-MDCK ER, Solubility).
Model Variants: Three BERT architectures: Tiny-BERT, Small-BERT, and Base-BERT.
Training Protocol:
- Pre-training: Masked Language Modeling (MLM) with dynamic masking (15% masking). No Next Sentence Prediction (NSP).
- Fine-tuning: 3-fold cross-validation with Bayesian hyperparameter search (50 architectures) for BERT; Grid search for classical ML baselines (LASSO, RF, SVM, XGB, LGBM).
- Hardware: Distributed training on NVIDIA A30, L40S, H200, and A100 GPUs.

Key Variables Investigated

Randomness: Impact of model weight initialization and data sampling seeds.
Standardization Noise: Simulating the mixing of PubChem-standardized SMILES with ChEMBL-standardized SMILES (which use different pipelines: OpenEye/OEChem vs. RDKit).
Tokenization: Comparison of WordPiece vs. Byte Pair Encoding (BPE).
Scaling: Systematic variation of Dataset Size (from 2.5% to 80% of PubChem) and Model Size.

3. Key Contributions

Systematic Ablation of Scaling Laws: The study provides the first rigorous numerical evidence suggesting that scaling laws (larger models + larger datasets = better performance) hold true for encoder-only CLMs with MLM objectives, provided data quality is consistent.
Identification of Standardization Noise: The authors identify standardization noise (mixing SMILES from different databases with different processing rules) as a primary cause of reported performance inconsistencies. They demonstrate that mixing protocols can severely degrade pre-training metrics and cause model divergence.
Benchmarking against Classical ML: A comprehensive comparison showing that well-tuned Base-BERT models generally outperform or match classical ML models on ADME tasks, but at a significantly higher computational cost.
Reproducibility: The authors released all code, data splits, pre-trained models, and fine-tuned artifacts to ensure transparency.

4. Key Results

A. Randomness and Tokenization

Randomness: Variations due to random seeds (initialization and data sampling) resulted in minor performance fluctuations ( $\approx 1\%$ ) compared to the impact of model size.
Tokenization: WordPiece and BPE showed comparable performance. WordPiece was selected for consistency with the original BERT, with no statistically significant difference observed in the small sample size of experiments.

B. The Impact of Standardization Noise (Critical Finding)

Performance Degradation: Introducing ChEMBL-standardized SMILES into the PubChem training set caused a sharp increase in validation loss and pseudo-perplexity (V-PPPL).
Divergence: In extreme cases (100% ChEMBL standardization in training), the Tiny-BERT model diverged and required early stopping.
Resilience: Larger models (Base-BERT) were more resilient to noise than Tiny-BERT, but performance still degraded significantly.
Conclusion: Mixing standardization protocols creates a "confused" vocabulary and distribution, hindering the model's ability to learn chemical syntax and semantics.

C. Scaling Laws (Dataset and Model Size)

Dataset Size: Pre-training performance (lower loss, higher accuracy) consistently improved as the dataset size increased from 2.5% to 80%.
- A "critical threshold" was observed around 10% of the data ( $\approx 12$ million samples), after which performance gains began to plateau for smaller models.
- Larger models (Base-BERT) were more sample-efficient, achieving high performance with smaller datasets compared to Tiny-BERT.
Model Size: Increasing model size (Tiny $\to$ Small $\to$ Base) consistently reduced validation loss and improved accuracy across all dataset sizes.
Fine-tuning: The trends held for downstream ADME tasks. Larger pre-training datasets and larger models led to higher Pearson $R$ and $R^2$ scores and lower MAE/RMSE.

D. Comparison with Classical ML

Performance: Base-BERT achieved superior or competitive results compared to LASSO, RF, SVM, XGB, and LGBM across all six ADME endpoints.
Cost-Benefit: While CLMs offer performance gains, they require massive computational resources (e.g., $\approx \$ 10,000$ for Base-BERT pre-training on AWS vs. minutes for SVM on a CPU). The authors suggest CLMs are an "economic" choice only when the task benefits from the general chemical language understanding of the foundation model.

5. Significance and Implications

Resolution of Literature Contradictions: The paper explains why previous studies might have reported contradictory results: many likely mixed datasets with inconsistent standardization protocols, introducing noise that masked the benefits of scaling.
Guidance for Future Research:
- Standardization is Paramount: Researchers must use a single, rigorous standardization pipeline (e.g., PubChem's) when curating pre-training corpora.
- Scaling is Valid: For MPP, "bigger is better" holds true for both model and dataset size, provided the data is clean.
- Resource Allocation: The study provides a cost-benefit analysis, helping practitioners decide when to invest in expensive CLM pre-training versus using faster, cheaper classical ML models.
Foundation for Scaling Laws: This work lays the groundwork for defining specific scaling laws for encoder-only chemical language models, moving the field from anecdotal evidence to systematic, data-driven understanding.

In summary, the paper argues that the perceived instability of CLMs in MPP is not an inherent flaw of the architecture but a result of data quality issues (standardization noise) and insufficient scaling. With proper data curation and sufficient scale, BERT-based models demonstrate robust, scalable performance for molecular property prediction.