🤖 machine learning

A Large Scale Investigation of Scaling Limits in Chemical Language Models

This large-scale study of over 30,000 experiments reveals that while scaling Chemical Language Models improves pretraining loss and chemical syntax understanding, these gains do not translate to proportional improvements in goal-directed molecular design, highlighting a critical disconnect between representation learning and downstream utility despite the introduction of the state-of-the-art NovoMolGen suite.

Original authors: Roshan Balaji, Kamran Chitsaz, Quentin Fournier, Nirav Pravinbhai Bhatt, Sarath Chandar

Published 2026-10-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Roshan Balaji, Kamran Chitsaz, Quentin Fournier, Nirav Pravinbhai Bhatt, Sarath Chandar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of modern science, there is a growing belief that if you simply make a computer model larger and feed it more data, it will inevitably become smarter. This idea, known as scaling, has transformed how we understand language, allowing machines to write essays, translate texts, and hold conversations by learning from massive libraries of human writing. Scientists have recently applied this same logic to chemistry, creating "chemical language models." These are artificial intelligence systems trained to read and write the strings of characters that represent molecules, much like a computer learns to read sentences. The hope has been that by making these chemical models bigger and training them on more molecules, they would naturally become better at designing new drugs from scratch, finding novel compounds that could cure diseases or treat ailments.

However, a recent large-scale investigation challenges this straightforward assumption. Researchers set out to test whether the rules that govern language learning also apply to the complex world of chemical design. They built a family of models ranging from very small to extremely large, training them on billions of molecules to see how their performance changed. The study, conducted by a team from institutions including IIT Madras and the Mila – Quebec AI Institute, reveals a surprising disconnect. While the models did indeed get better at understanding the basic grammar of chemistry as they grew, this improvement did not translate into better results when asked to design specific, useful drugs. In fact, the researchers found that a relatively small model could perform just as well as a massive one when tasked with the difficult job of finding new medicines, suggesting that simply making the computer bigger is not the key to unlocking the secrets of drug discovery.

The researchers began by training over thirty thousand different versions of these chemical models. They varied the size of the models from half a million parameters to one billion parameters, fed them different types of molecular data, and tested them using various methods of representing chemical structures. One key finding was that the models did learn to predict the next part of a chemical string with increasing accuracy as they grew larger and saw more data. This is similar to how a child learning a language eventually becomes better at guessing the next word in a sentence. The models became excellent at understanding the rules of chemical syntax, knowing which combinations of atoms form valid molecules and which do not. This internal understanding of chemical structure continued to improve even for the largest models, suggesting that the computer was successfully memorizing the "vocabulary" and "grammar" of chemistry.

Yet, when the researchers tested these models on the actual goal of drug design, the story changed dramatically. They asked the models to generate new molecules that would bind to specific disease targets or meet a set of desirable chemical properties, a process that involves a complex loop of trial and error. Here, the benefits of size vanished. The performance of the models in designing these goal-directed molecules plateaued, or flattened out, once the models reached a size of about five million parameters. Increasing the model size to one billion parameters, which is two hundred times larger, yielded almost no additional improvement in the quality of the drugs they could design. A model with five million parameters performed just as well as the massive one-billion-parameter model in finding new candidates for drug discovery.

This result suggests that the current method of training these models, which focuses on predicting the next character in a chemical string, teaches the computer the rules of chemistry but not necessarily the deeper meaning of how those molecules interact with biological systems. The models learned to speak the language of chemistry fluently, but they did not learn to use that language to solve specific problems. The researchers found that while the larger models could generate valid molecules, they did not become significantly better at finding the specific, high-quality molecules needed for medicine. The study indicates that the bottleneck is not the size of the model or the amount of data, but rather the way the models are trained and the objectives they are asked to optimize.

The implications of this discovery are significant for the future of artificial intelligence in science. It suggests that the path forward for drug discovery may not lie in building ever-larger computer models, but in developing smarter ways to train the models we already have. The researchers propose that instead of just teaching computers to predict the next chemical symbol, we need to teach them to understand the functional properties of molecules, such as how they interact with proteins or how they behave in the human body. By focusing on these deeper chemical meanings rather than just the surface-level patterns, scientists might be able to create more efficient and effective tools for designing the next generation of life-saving medicines. The work serves as a reminder that in the complex world of chemistry, bigger is not always better, and that true intelligence requires more than just a vast memory of patterns.

Technical Summary: A Large Scale Investigation of Scaling Limits in Chemical Language Models

Problem Statement

Chemical Language Models (CLMs) have increasingly adopted the scaling paradigm of Natural Language Processing (NLP), pretraining autoregressive transformers on massive molecular corpora with the implicit assumption that minimizing next-token cross-entropy directly translates to improved molecular design. However, the relationship between design choices, training dynamics, and downstream goal-directed generation quality remains poorly understood. While recent studies have explored scaling in CLMs, they often fail to establish how pretraining gains translate to practical molecular optimization or account for the costs of downstream optimization. This paper investigates whether continued improvements in pretraining loss and internal chemical representations necessarily yield comparable gains in goal-directed molecular design.

Methodology

The authors present a compute-controlled scaling study involving over 30,000 experiments across a wide range of variables:

Model Scales: Models ranging from 0.5M to 1B parameters.
Representations & Tokenization: Three molecular string representations (SMILES, SELFIES, SAFE) and two tokenization strategies (atom-level and byte-pair encoding).
Architectures: Both decoder-only (Qwen3 backbone) and encoder–decoder (T5Gemma backbone) models.
Datasets: Four leakage-controlled corpora of varying size and diversity: MOSES, ChEMBL, PubChem, and ZINC-22. To prevent data leakage, molecules with high Tanimoto similarity to known actives and benchmark targets were removed from all training corpora.
Evaluation Metrics:
- Pretraining: Validation set negative log-likelihood measured in bits per molecule (normalizing for sequence length differences) to enable cross-representation comparison.
- Scaling Analysis: IsoFLOP profiling to determine compute-optimal model sizes ( $N_{opt}$ ) and data sizes ( $D_{opt}$ ) at fixed compute budgets ( $C \approx 6ND$ ).
- Internal Representations: Layer-wise linear probing for 36 molecular properties (physicochemical, PMO objectives, docking scores) and validity. Sparse Autoencoder (SAE) analysis to measure feature monosemanticity and property correlation.
- Downstream Performance: Goal-directed generation evaluated on the Practical Molecular Optimization (PMO) benchmark (22 tasks, 10k oracle calls) and structure-based docking (5 targets, 3k calls). Fine-tuning was performed using the REINVENT framework with Augmented Hill-Climb reinforcement learning under fixed oracle budgets.

Key Contributions

NovoMolGen Suite: The release of a family of 42 CLMs pretrained from scratch, establishing a new baseline for chemical language modeling.
IsoFLOP Profiles for Chemistry: The first application of IsoFLOP profiling to chemical language models, establishing power-law scaling relationships for chemical sequence modeling.
Decoupling of Pretraining and Design: A comprehensive empirical demonstration that while pretraining loss and internal chemical representations improve with scale, goal-directed molecular generation performance saturates early.
Representation and Corpus Analysis: Systematic comparison showing that molecular representation (SMILES vs. SELFIES vs. SAFE) and corpus diversity have a larger impact on loss than tokenization or architecture.
Interpretability Insights: Evidence that chemical syntax is learned early in training, while semantic properties (functional groups, docking scores) emerge later and become more accessible with scale, yet this does not linearly translate to downstream optimization gains.

Results

1. Pretraining Scaling Laws

Chemical sequence modeling follows compute-optimal scaling trends similar to natural language. The study found that the optimal model size scales as $N_{opt} \propto C^{0.69}$ and optimal data size as $D_{opt} \propto C^{0.31}$ . Validation set loss (bits per molecule) decreases smoothly with increasing compute across all model sizes (0.5M to 1B).

2. Impact of Design Choices

Representation: SMILES consistently achieved the lowest bits per molecule across all scales, outperforming SELFIES and SAFE. While SELFIES tokens are individually easier to predict, the longer sequence length required to encode molecules resulted in higher per-molecule loss.
Corpus: The pretraining corpus had the most significant effect on loss. MOSES (narrow, drug-like) yielded the lowest loss, while ZINC-22 (broad, make-on-demand) yielded the highest, reflecting the difficulty of modeling diverse chemical spaces.
Tokenization & Architecture: These factors had minor, scale-dependent effects on loss compared to representation and corpus.

3. Internal Representations

Syntax vs. Semantics: Linear probing revealed a distinct learning trajectory. Chemical syntax (validity) saturates early (reaching 90% of final gain after 0.1 tokens per parameter). In contrast, semantic properties (docking scores, PMO objectives, physicochemical descriptors) emerge more slowly and continue to improve with scale and training tokens.
SAE Analysis: Sparse Autoencoders showed increasing monosemanticity and feature-property correlations as training progressed, indicating that chemical features become more linearly accessible in the latent space.

4. Downstream Goal-Directed Generation

Despite continuous improvements in pretraining loss and internal representations, goal-directed generation performance saturates at approximately 5M parameters.

PMO Benchmark: A 5M-parameter NovoMolGen model outperformed REINVENT and f-RAG. Scaling up to 1B parameters yielded only marginal gains (total PMO score increased from 16.45 to 16.68).
Docking: Similarly, docking scores for novel hits plateaued beyond 5M parameters, with the total docking score improving only from -65.8 to -66.2 kcal/mol.
Unconstrained Generation: Scaling did not improve generalization in unconstrained sampling; Fréchet ChemNet Distance (FCD) did not consistently decrease with lower loss, and sampled molecules remained anchored to the training distribution.

Significance and Claims

The paper claims a fundamental disconnect between chemical representation learning and downstream molecular design under current self-supervised string objectives. The authors argue that:

Diminishing Returns: Simply scaling parameter counts under current pretraining objectives yields diminishing returns for practical drug discovery tasks.
Efficiency: Compact CLMs (~5M parameters) preserve competitive downstream performance while drastically reducing the computational cost of generation and policy updates (approx. 200x cheaper per policy update compared to 1B models).
Limitations of Current Benchmarks: Current benchmarks and pretraining objectives (focused on chemical validity and syntax) fail to capture the complexity of chemistry required for functional design. The dominant signal in large chemical libraries (PubChem, ZINC-22) teaches syntax rather than functional semantics.
Future Directions: The authors motivate the development of pretraining paradigms that more directly learn chemical semantics, such as incorporating protein-ligand interaction signals, physicochemical properties, or reinforcement learning as a "fitness" signal earlier in training. They suggest that future progress requires objectives beyond chemical validity rather than simply larger models.

The authors conclude that while NLP-inspired scaling laws hold for pretraining loss and internal representations, they do not hold for goal-directed molecular generation, suggesting that the field must shift focus from scaling model size to improving sample efficiency and learning objectives.