💬 NLP

In-Place Tokenizer Expansion for Pre-trained LLMs

This paper introduces an in-place tokenizer expansion method that upgrades pre-trained LLMs by continuing BPE merges on a multilingual corpus and initializing new embeddings via sub-token averaging, thereby significantly reducing token counts and improving decode speed for languages like Hindi and Vietnamese without sacrificing model quality.

Original authors: Jimmy T. H. Smith, Tarek Dakhran, Alberto Cabrera, Simon S. Lee, Paul Pak, Aditya Tadimeti, Tim Seyde, Maxime Labonne, Alexander Amini, Mathias Lechner

Published 2026-07-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jimmy T. H. Smith, Tarek Dakhran, Alberto Cabrera, Simon S. Lee, Paul Pak, Aditya Tadimeti, Tim Seyde, Maxime Labonne, Alexander Amini, Mathias Lechner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Language of Machines: A Quick Primer

Imagine you are teaching a robot to speak. To do this, you can't just give it a dictionary of every word in the universe; that would be too heavy to carry. Instead, you give it a set of building blocks. In the world of Artificial Intelligence, these blocks are called tokens. Think of tokens as the smallest chunks of text a computer understands—sometimes a whole word like "cat," but often just a syllable like "cat" or even a single letter like "c."

When a robot learns a language, it decides how to break sentences into these blocks. If it learns English first, it might have a special block for the word "the." But if you later ask it to speak Hindi or Vietnamese, and it wasn't prepared for those languages, it might have to break those words into tiny, inefficient pieces, like "H-i-n-d-i" instead of "Hindi." This is like trying to build a house with a hammer when you really need a screwdriver; it works, but it's slow, clunky, and wastes energy. This paper tackles the problem of how to upgrade a robot's "toolbox" (its vocabulary) after it has already been trained, without having to rebuild the whole robot from scratch.

The Puzzle of the Broken Toolbox

Meet Jimmy and his team at Liquid AI. They were working on a very smart robot called LFM2, which lives on devices like phones and laptops. This robot was great at English and coding, but when it tried to speak languages like Hindi, Vietnamese, or Thai, it stumbled. Why? Because its vocabulary was like a toolbox that had been packed for a trip to London, not Bangkok. When the robot tried to write "Hindi," it didn't have a single block for the whole word. Instead, it had to use four or five tiny blocks to piece it together. This made the robot slow, tired, and hungry for battery power.

The team faced a tough choice. They could throw away their hard work, build a brand new robot from scratch with a better toolbox, and start over. Or, they could try to sneak a new set of tools into the old robot without breaking it. The second option is like trying to add a new drawer to a fully assembled, running car engine while it's still driving down the highway. Most people thought this was impossible or too risky.

The "In-Place" Upgrade

The team discovered a clever way to do exactly that. They called it Tokenizer Expansion. Instead of throwing away the old toolbox, they kept the original tools and simply added new ones to the end of the box.

Here is how they did it, step-by-step:

The Smart Copy-Paste: They took the robot's existing list of building blocks and asked it to keep learning on a mix of many different languages. As the robot learned new words, it didn't throw away the old blocks. Instead, it created new, bigger blocks for words like "Hindi" or "Thai." Crucially, every new block was built by snapping together the old, smaller blocks it already knew. It was like realizing that if you have a "H" block and an "i" block, you can snap them together to make a "Hi" block, and then snap that with "nd" and "i" to make "Hindi."
The Magic Glue (Embedding Initialization): Now the robot had new blocks, but it didn't know what they meant. The team had to teach the robot what these new blocks represented. They used a simple but brilliant trick: they looked at the old blocks that made up the new one and gave the new block the "average" meaning of its parts. If a new block was made of three old blocks, the robot's brain gave it the average personality of those three. This meant the robot didn't have to guess; it started with a very good guess.
The Two-Stage Workout: The robot wasn't ready to run a marathon yet. If they let it run immediately, it would trip. So, they did a two-stage workout:
- Stage 1: They froze the robot's brain (everything except the new blocks) and only let the new blocks learn. This was like letting the new drawers in the toolbox get used to holding things without moving the rest of the car.
- Stage 2: Once the new blocks were comfortable, they unfroze the whole robot and let it practice reading and writing in many languages again. This helped the robot understand how the new blocks fit into the big picture.

The Results: Faster, Smarter, and Lighter

The results were impressive. By using this new, expanded toolbox, the robot could speak Hindi, Vietnamese, and Thai much more efficiently.

Hindi text was broken into 2.4 times fewer blocks.
Vietnamese was 2.6 times fewer.
Thai saw the biggest jump, needing 4.0 times fewer blocks.

Because the robot had to process fewer blocks to say the same thing, it became much faster. The team estimated that on a phone, speaking these languages could be 2.2 to 3.7 times faster per character. It was like switching from walking to a bicycle; the robot covered the same distance in a fraction of the time.

What They Learned (and What They Avoided)

The team also learned what not to do, which is just as important.

Don't touch the old tools: In one experiment, they tried to let the robot re-learn the meanings of its old blocks while adding the new ones. This made the robot confused and it started repeating itself like a broken record. They found that keeping the old blocks exactly as they were was the secret to success.
Don't just read English: When they tried to teach the robot the new blocks using mostly English text, the robot got good at answering multiple-choice questions but terrible at actually writing sentences. It could pick the right answer but couldn't tell a story. They learned they had to feed it a balanced diet of many languages to keep it healthy.

The Bottom Line

This paper shows that you don't always need to build a new robot to make it speak a new language. If you are the one who built the robot, you can upgrade its vocabulary right in the middle of its life. By carefully adding new tools and giving them the right "average" meaning, you can make the robot faster and more efficient for everyone, without losing the skills it already had. It's a practical recipe for making AI more inclusive and faster, right on the device in your pocket.

Technical Summary: In-Place Tokenizer Expansion for Pre-trained LLMs

Problem Statement
Tokenizers for Large Language Models (LLMs) are typically fixed at the start of pre-training, with vocabulary allocation reflecting the distribution of the initial corpus. As deployment priorities shift to include underrepresented languages, these languages suffer from high "token fertility"—they are split into significantly more tokens per word compared to languages the tokenizer was optimized for. This fragmentation increases decode latency, compute, and energy consumption, particularly for on-device models where the embedding and LM-head matrices constitute a material share of per-token decode bandwidth. While cloud models can afford broad vocabularies, edge models must ship with compact vocabularies, accepting fragmentation outside a fixed language set. Standard remedies, such as pre-training from scratch with a new tokenizer or using zero-shot transfer methods to arbitrary third-party tokenizers, are either computationally wasteful or solve a harder problem than necessary.

Methodology
The authors propose Tokenizer Expansion, an in-place recipe for upgrading a pre-trained model's tokenizer when the model producer controls the design. The approach consists of three main components:

Tokenizer Construction (Continued-BPE):
Instead of training a new tokenizer independently or taking a union of vocabularies, the authors continue the Byte-Pair Encoding (BPE) merge training of the source tokenizer on a multilingual corpus weighted toward under-tokenized languages.
- Properties: Existing merges are held fixed; new merges are appended. This ensures a deterministic mapping where most source tokens carry over as single tokens, and every new token has an exact decomposition into one or more source tokens.
- Scale: Applied to an 8B-parameter Mixture-of-Experts (MoE) model, expanding the vocabulary from 65K to 128K active entries.
Embedding Initialization:
The embedding matrix is reinitialized to match the new vocabulary size ( $|V_{new}|$ ).
- Carried-over tokens: Rows for tokens that map 1-to-1 from the source are copied unchanged.
- New tokens: Rows for new tokens are initialized as the mean of the embedding rows of their constituent source sub-tokens. This ensures the new rows have a norm comparable to existing rows without requiring random initialization.
Two-Stage Adaptation:
To recover the quality of the source checkpoint, the model undergoes a specific training pipeline:
- Stage 1 (Embedding-Only Training): The model body is frozen. Only the new embedding rows (and their tied LM-head counterparts) are trained for 600B tokens on a multilingual mixture focused on under-tokenized languages. Crucially, the rows copied from the source are frozen to prevent them from drifting away from the frozen body.
- Stage 2 (Full Continued Pre-Training): All parameters are unfrozen. The model continues pre-training for 400B tokens on a balanced multilingual corpus. This stage is critical for re-balancing the model and integrating the new vocabulary without regressing performance on previously supported languages.
- Mid-Training/Post-Training: The adapted checkpoint proceeds through standard mid-training (context extension to 128K) and post-training phases.

Key Results
The method was applied to the LFM2-8B-A1B checkpoint to produce LFM2.5-8B-A1B.

Token Efficiency: The expanded tokenizer significantly reduces token counts for under-tokenized languages while maintaining parity for English and code.
- Reductions: Hindi (~~2.4× fewer tokens), Vietnamese (~~2.6×), and Thai (~4.0×).
Quality Recovery:
- Zero-shot Swap: Simply swapping the tokenizer caused a drop of ~5.8 aggregate benchmark points.
- Stage 1: Recovered ~4.8 points, largely restoring English and code performance.
- Stage 2: Closed the remaining gap and surpassed the source model by 3.6 aggregate points, matching or exceeding the source on Knowledge, Math, Code, and Multilingual categories.
- Multilingual Benchmarks: Previously supported languages (e.g., Arabic, German, Japanese) maintained or slightly improved performance, while under-tokenized languages saw substantial gains (e.g., Vietnamese +11.6 points, Hindi +7.6 points on Global-MMLU).
On-Device Latency:
- Trade-off: A larger vocabulary increases per-token decode cost (measured as a ~7–10% drop in tokens/sec on reference devices due to memory bandwidth constraints).
- Net Speedup: When combining token compression with the per-token cost, the expanded tokenizer yields a 2.2× to 3.7× per-character decode speedup for Thai, Bengali, Vietnamese, and Hindi across reference devices (M4 Max, Snapdragon 8 Elite).
- Regression: For languages with near-parity compression (English, code, major European languages), the per-character decode time regresses by up to ~9%.

Negative Findings and Ablations
The paper reports two critical negative findings that shaped the final recipe:

Training the Full Embedding in Stage 1: Training all embedding rows (including copied source rows) while the body is frozen caused a degradation in generative capabilities (e.g., MATH500, HumanEval+ collapse) and severe repetition, even though multiple-choice accuracy remained intact. Restricting training to only new rows was necessary.
Stage 2 Data Mixture: Using a predominantly English mixture for Stage 2 continued pre-training caused a "generative collapse" (loss of reasoning and multilingual generation) despite maintaining or improving multiple-choice scores. A balanced multilingual mixture was required to preserve generative quality.

Significance and Claims
The authors claim this is the first account of a model-side recipe for in-place, continued-BPE tokenizer expansion at production scale (specifically for an 8B on-device MoE).

Efficiency: It demonstrates that vocabulary expansion can be achieved without discarding accumulated pre-training compute, provided the tokenizer is constructed as a continuation of the source merges.
Deployment: It offers a practical solution for on-device models to support additional languages efficiently, trading a small uniform per-token cost for massive efficiency gains on specific scripts.
Evaluation Insight: The work highlights that standard multiple-choice benchmarks (like MMLU) may fail to detect degradation in generative capabilities during vocabulary adaptation, necessitating the inclusion of generative benchmarks in evaluation.

The authors release the model weights (LFM2.5-8B-A1B) and the expanded tokenizer, noting that the recipe is specifically designed for settings where the model producer controls the tokenizer upgrade, rather than for adapting to arbitrary third-party tokenizers.