Evaluating the Impact of Tokenization Schemes on the Performance of Chemical Language Models
This study presents the first systematic evaluation of nine SMILES tokenization strategies across three model scales, revealing that while rule-based tokenizers excel in generative validity, corpus-based and hybrid approaches outperform in downstream property prediction, ultimately recommending a hybrid strategy for generalizable chemical language models.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern laboratory, the language of chemistry is increasingly written not just in test tubes and beakers, but in lines of text. Scientists have long used a shorthand called SMILES to represent complex molecules as simple strings of characters, much like a sentence describes a story. This format allows powerful computer programs, originally designed to read human language, to analyze chemical structures. These programs, known as chemical language models, can predict how a new drug might behave or even invent entirely new molecules from scratch. However, for a computer to understand a chemical string, it must first break that string down into smaller, manageable pieces, a process called tokenization. Just as a reader breaks a sentence into words to understand its meaning, a computer breaks a chemical string into tokens to grasp the structure of the molecule. The critical question facing researchers is how best to perform this breakdown: should the computer follow strict, pre-written rules about what constitutes a chemical part, or should it learn to find patterns on its own by reading millions of examples?
A team of researchers at the University of Kaiserslautern-Landau and the German Research Centre for Artificial Intelligence set out to answer this question with a comprehensive study. They examined the history of the field, reviewing over two hundred research papers to identify the different ways scientists have been teaching computers to read chemical strings. From this vast survey, they selected nine distinct methods and organized them into three groups: those that rely on fixed chemical rules, those that learn entirely from data, and those that mix both approaches. To test these methods fairly, the researchers built a series of computer models of varying sizes, ranging from small to very large, and trained them using the same massive dataset of nearly two million chemical strings. They then put these models through two different kinds of tests. First, they checked how well the models could reconstruct the original chemical strings after hiding parts of them, a measure of how well the computer learned the basic grammar of chemistry. Second, they tested the models on real-world tasks, asking them to predict specific properties of molecules, such as how well a drug might dissolve in water or how likely it is to cross the blood-brain barrier.
The results revealed a clear and surprising trade-off between the ability to generate new molecules and the ability to predict their properties. The models trained with rule-based tokenizers, which strictly followed chemical definitions, proved to be the best at reconstruction. They could piece together the hidden parts of a chemical string with high accuracy and, most importantly, ensured that the resulting molecules were chemically valid and physically possible. This makes them the superior choice for tasks where the goal is to invent new compounds, as they rarely produce impossible or nonsensical results. However, when the researchers turned to the task of predicting molecular properties, the story changed. The models that used data-driven or hybrid approaches, which allowed the computer to learn flexible patterns from the data itself, consistently outperformed the rigid rule-based models. These flexible models learned to recognize subtle relationships within the chemical strings that the strict rules missed, leading to more accurate predictions of how a molecule would behave in the real world.
The study also explored whether simply making the computer models larger would solve the problem. They tested models with 7 million, 50 million, and 150 million parameters to see if more computing power could overcome a poor choice of tokenization. They found that while larger models generally performed better, the gains diminished significantly as the models grew from 50 million to 150 million parameters. This suggests that the way the chemical data is broken down is often more important than the sheer size of the computer model. A well-chosen tokenization strategy can yield better results than simply adding more computing capacity. Ultimately, the researchers concluded that there is no single best method for all situations. If the goal is to design new molecules, a rule-based approach is the safest and most effective path. But if the goal is to predict how a molecule will act, a hybrid approach that combines chemical knowledge with data-driven flexibility offers the most powerful solution. This work provides a clear guide for scientists, showing that the choice of how to teach a computer to read chemistry is just as critical as the computer itself.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.