💬 NLP

Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES

This study demonstrates that Byte-Pair Encoding (BPE) and Unigram-LM produce fundamentally distinct and near-disjoint subword vocabularies for chemical SMILES, revealing that the choice of tokenization algorithm is a critical modeling decision rather than a neutral default.

Original authors: Hunter Heidenreich

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hunter Heidenreich

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: How Do We Teach Computers to Read Chemical Formulas?

Imagine you are trying to teach a computer to understand the language of chemistry. Chemists write molecules using a code called SMILES (a string of letters and symbols like CC(=O)Oc1ccccc1).

Before a computer can learn anything, it needs a tokenizer. Think of a tokenizer as a translator that breaks a long sentence into smaller, manageable chunks (words or sub-words) that the computer can understand.

For years, the field of chemistry has blindly copied a method from natural language processing (like how computers read English) called BPE (Byte-Pair Encoding). The researchers in this paper asked a simple question: "Is this the best way to cut up chemical formulas, or is there a better way?"

They compared the standard method (BPE) against a different method called Unigram-LM.

The Experiment: Two Different Scissors

Imagine you have a long, complex necklace made of different colored beads (the chemical formula). You need to cut this necklace into smaller pieces to study it. You have two different pairs of scissors:

The Greedy Scissors (BPE): These scissors look for the two beads that appear next to each other the most often in the whole pile of necklaces. They glue those two together into a single "super-bead" and repeat the process. They are aggressive and greedy, trying to make big chunks out of frequent patterns.
The Probabilistic Scissors (Unigram-LM): These scissors start with a huge pile of tiny beads and ask, "If I remove this specific bead, how much does it hurt the overall picture?" They carefully prune away pieces that aren't essential, keeping a more granular, detailed view.

The Surprising Discovery: They Don't Agree

The researchers expected that because chemical formulas are very strict (atoms must connect in specific ways), both scissors might end up cutting the necklace in almost the same places. They thought the "rules of chemistry" would force the two methods to converge.

They were wrong.

Even when they used the exact same chemical data, the exact same starting rules, and the exact same vocabulary size, the two methods produced completely different sets of "words."

The Overlap is Tiny: If you took the list of "super-beads" created by BPE and compared it to the list created by Unigram-LM, they shared almost nothing. In the worst case, they shared less than 16% of their pieces. In the best case (looking at the most important, high-frequency pieces), they shared less than 5%.
The "Cut" Depth: The two methods agreed on where to make the cuts (the skeleton of the molecule), but they disagreed on how deep to cut.
- BPE tended to make coarser cuts. It would glue whole rings of atoms together into one big token.
- Unigram-LM made finer cuts. It kept the rings broken down into smaller, near-atomic pieces.
- Result: Unigram-LM ended up using 29% to 41% more tokens (pieces) to describe the same molecule than BPE did.

A Visual Analogy: The Map vs. The Street View

Imagine you are looking at a map of a city.

BPE is like a map that groups entire neighborhoods into single blocks. It says, "This whole area is 'Downtown'." It's efficient and uses fewer words.
Unigram-LM is like a street view that lists every single building and street corner. It says, "Here is a bakery, here is a park, here is a house." It uses many more words but gives a more detailed breakdown.

The paper found that these two maps are not interchangeable. If you switch from one to the other, the computer sees a fundamentally different structure of the data.

Key Findings in Plain English

It's a Design Choice, Not a Default: The field has been using BPE by default because it's what natural language models use. This paper proves that in chemistry, you cannot just copy-paste that choice. You have to actively decide which "scissors" to use because they produce different results.
The Difference is Stable: This disagreement didn't happen just by chance. It happened across different types of chemicals (common drugs, rare natural products, and diverse molecules) and even when they changed the rules about how to handle complex atoms.
Scale Doesn't Fix It: Usually, if you give a computer more data or a bigger vocabulary, things might smooth out. The researchers tested this by making the vocabulary 8 times larger. The two methods still didn't agree. They remained distinct.
The "Nesting" Effect: Even though they use different words, they don't fight each other. The Unigram-LM cuts are almost always "inside" the BPE cuts. It's like BPE says "Cut the whole pizza," and Unigram-LM says "Cut the pizza, then cut the slices." They are compatible, just at different levels of detail.

What This Means for the Reader

The paper concludes that the algorithm you choose is a major modeling decision.

If you choose BPE, you get shorter sequences (fewer tokens), which is cheaper for the computer to process, but you lose some granular detail about the molecule's structure.
If you choose Unigram-LM, you get a more detailed, fine-grained view of the molecule, but it requires the computer to process more tokens.

The authors did not test which one makes the computer smarter at predicting chemical reactions or drug properties. They only proved that the two methods create different languages. Therefore, the chemistry community can no longer assume they are the same; they must choose their "scissors" carefully.

Technical Summary: Where to Cut, How Deep: BPE and Unigram-LM on Chemistry SMILES

Problem Statement
Chemical language models operating on SMILES strings universally rely on tokenizers to map molecular strings to integer IDs, a foundational design choice that dictates vocabulary, operational granularity, and effective sequence length. The field has largely inherited Byte-Pair Encoding (BPE) from natural language processing (NLP) without rigorous scrutiny. While Unigram-LM is the principal alternative in NLP, known for building structurally different vocabularies, its behavior in chemistry remains unverified. A "convergence null" hypothesis exists: because chemistry SMILES operates on a tiny, valence-constrained alphabet (158 glyphs) where structural constraints sharply concentrate valid sequences, frequency-based algorithms like BPE and Unigram-LM might be forced to converge on the same vocabulary. This study investigates whether the choice of subword algorithm remains a distinct modeling decision in chemistry or if the chemical domain's constraints drive these algorithms to produce interchangeable vocabularies.

Methodology
The authors conducted a controlled comparison of BPE and Unigram-LM across 22 matched conditions, isolating the algorithm as the sole variable.

Fixed Base: Both algorithms were trained on a fixed 165-token OpenSMILES glyph base (Smirk), ensuring complete coverage of conformant molecules and eliminating vocabulary coverage as a confounding factor.
Experimental Grid: The study spanned three corpus typologies (diverse: PubChem; drug-like: ZINC-22; natural-products: COCONUT), two pre-tokenization boundary policies (merging across bracketed atoms vs. not), and three vocabulary sizes ( $V \in \{256, 512, 1024\}$ ). A sensitivity block extended to $V=2048$ and $V=8192$ .
Metrics: The authors measured three direct contrasts:
1. Membership: Vocabulary overlap (Jaccard index) of learned multi-glyph pieces.
2. Granularity: Fertility (mean tokens per molecule on held-out data).
3. Distribution: Token-frequency imbalance.
- Additionally, mechanism diagnostics were computed, including dead-zone surplus, whole-pretoken absorption, scaffold fraction, and segmentation entropy.
Learnability Gate: Conditions were filtered based on the "learnability bar" (Gowda and May), ensuring token embeddings could be reliably trained (95% of learned pieces firing at least 100 times).

Key Results
The study finds that BPE and Unigram-LM do not converge in the chemistry domain; they produce measurably different vocabularies across all tested conditions.

Near-Disjoint Vocabularies (Membership): The overlap between the learned subword pieces of the two algorithms is minimal. The unweighted Jaccard overlap never exceeds 0.161, and the frequency-weighted overlap (reflecting pieces a model actually updates) never exceeds 0.05. The algorithms disagree most significantly on high-frequency pieces: BPE tends to merge aromatic rings and chains into single tokens, while Unigram-LM remains near-atomic, often fragmenting these structures.
Systematic Granularity Gap: Unigram-LM consistently segments molecules into 29–41% more tokens than BPE. BPE acts as a "strict coarsening" of Unigram-LM's segmentation on 80–99% of molecules; the algorithms largely agree on where to cut but not how deeply.
Stability and Robustness: The divergence holds across corpus typologies, boundary policies, and vocabulary sizes. Even at $V=8192$ (eight times the headline size, beyond the learnable regime), the overlap and granularity gaps remain essentially unchanged. The divergence is not an artifact of small vocabulary sizes or specific corpus characteristics.
Mechanism: The difference is driven by algorithmic mechanics. BPE's bottom-up greedy merging creates compositional closure (pieces concatenate existing pieces), while Unigram-LM's top-down pruning retains long heteroatom chains that cannot be decomposed into in-vocab parts. BPE achieves higher whole-pretoken absorption, whereas Unigram-LM exhibits higher segmentation entropy.
Generalization: The granularity gap persists off-domain. When tokenizers trained on one corpus are applied to others (including transition-metal complexes and cyclic peptides), BPE maintains near-native sequence lengths, while Unigram-LM shows modest domain sensitivity but retains its finer segmentation.

Significance and Claims
The paper claims that the subword algorithm is a modeling decision, not a free default.

Dislodging the Default: The field's inherited assumption that BPE and Unigram-LM are interchangeable at the tokenizer level is incorrect for chemistry. Swapping the algorithm changes which substructures become atomic embedding units and alters the effective sequence length by roughly one-third.
Algorithm vs. Corpus: The choice of algorithm has a larger impact on tokenizer structure than the choice of training corpus. Tokenizers transfer across domains with near-native efficiency, suggesting the algorithm defines the tokenizer's behavior more than the data it is trained on.
Scope: This is a tokenizer-level study. The authors do not claim which algorithm yields better downstream language model performance, noting that intrinsic separation does not necessarily translate to downstream gaps. However, the study establishes the precondition for such comparisons by proving the resulting vocabularies are not interchangeable.
Limitations: The study acknowledges that the "morphological alignment" metric from NLP does not transfer to SMILES due to the lack of a gold-standard segmentation. It also notes that the divergence direction (which algorithm is more compressive) may be domain-specific, as protein sequences show the opposite trend.

In summary, the paper demonstrates that the small, valence-constrained alphabet of chemistry does not force BPE and Unigram-LM to converge. Instead, the algorithms build near-disjoint vocabularies with distinct structural properties, making the selection of the subword algorithm a critical, non-neutral design choice for chemical language models.