💬 NLP

LiteToken: Removing Intermediate Merge Residues From BPE Tokenizers

This paper introduces LiteToken, a method that identifies and removes infrequent "residue" tokens from BPE vocabularies to reduce model parameters and improve robustness against noisy inputs, often without requiring additional fine-tuning of pretrained models.

Original authors: Yike Sun, Haotong Yang, Zhouchen Lin, Muhan Zhang

Published 2026-02-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yike Sun, Haotong Yang, Zhouchen Lin, Muhan Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to read and write human language. To do this, you first have to break sentences down into small chunks called "tokens" (like words or parts of words) that the robot can understand. The most popular way to do this is a method called BPE (Byte Pair Encoding).

Think of the BPE process like a construction crew building a vocabulary from scratch. They start with individual letters. Then, they look at a massive library of books and say, "Hey, the letters 't' and 'h' appear together all the time. Let's glue them into a new block called 'th'." Later, they see 'th' and 'e' together often, so they glue those to make 'the'. They keep doing this, building bigger and bigger blocks.

The Problem: The "Scaffolding" Left Behind

The paper identifies a messy side effect of this construction process. Sometimes, the crew builds a temporary block (like "includ") because it was very common at that moment in the construction phase. But later, they build an even bigger block ("include" or "including").

In a perfect world, the crew would knock down the temporary "includ" block and throw it away. But in the current BPE method, they leave it standing in the vocabulary list anyway.

The authors call these leftover blocks "Intermediate Merge Residues."

The Analogy: Imagine you are building a house. You use temporary scaffolding to reach the second floor. Once the roof is on, you take the scaffolding down. But in this scenario, the construction crew forgot to remove the scaffolding. Now, your house is full of useless wooden poles that take up space, look ugly, and confuse anyone trying to walk through the rooms.

These "scaffolding tokens" (like "includ", "delet", "framewor") are rarely used in real life because the full words ("include", "delete", "framework") are preferred. However, they still sit in the robot's dictionary, taking up memory and processing power. Worse, because the robot barely ever sees them, it gets confused when it does encounter them (like when someone misspells a word or a hacker tries to trick the system).

The Solution: LiteToken

The authors created a tool called LiteToken to clean up this mess. It's like a "spring cleaning" service for the robot's dictionary.

Here is how it works, step-by-step:

The Detective Work: The tool scans the dictionary and looks for those "scaffolding" tokens. It asks two questions:
- Is this token rarely used on its own? (Low frequency).
- Does this token only ever appear in very specific, predictable situations? (Low "entropy" or variety).
- Example: If the token "includ" only ever appears right before "e" or "ing", it's probably just a leftover piece. But if a token like "re" appears in "rewrite," "recover," and "refund," it's a useful building block, so it stays.
The Removal: Once identified, these useless tokens are marked for removal.
The Re-assembly: If the robot encounters a word that used to be split by these useless tokens, the tool breaks the word down into its basic letters and then re-builds it using only the good, useful blocks. It's like taking down the scaffolding and replacing it with a clean, solid wall.

The Results: A Lighter, Stronger Robot

The paper tested this on several famous AI models (like Qwen, Llama, and GPT). Here is what they found:

It's "Plug-and-Play": You don't need to retrain the robot or teach it anything new. You can just swap the old dictionary for the new, cleaned-up one, and the robot works just as well.
Saves Space: By removing about 5% to 10% of the useless tokens, the robot needs less memory and performs calculations faster. It's like removing heavy, useless bricks from a backpack so you can run faster.
Better at Handling Mistakes: When people make typos or try to trick the robot with weird inputs, the old robot often gets confused because it tries to force the "scaffolding" tokens to make sense. The new "Lite" robot handles these errors much better because it doesn't rely on those fragile, half-built blocks.
No Loss in Smarts: Despite removing these tokens, the robot's ability to answer questions or write stories didn't get worse. It stayed just as smart.

Summary

In short, the paper argues that many AI models are carrying around a lot of "digital junk"—useless, half-finished word parts that were left over from their training. LiteToken is a simple tool that sweeps this junk out, making the AI models lighter, faster, and more robust against errors, all without needing to retrain them.

Technical Summary: LiteToken: Removing Intermediate Merge Residues From BPE Tokenizers

1. Problem Statement

While Large Language Models (LLMs) have seen dramatic growth in parameters and training data, the design of their tokenizers—specifically the widely used Byte Pair Encoding (BPE)—has received comparatively less scrutiny. This paper identifies a specific inefficiency in standard BPE vocabularies termed "intermediate merge residues" (also referred to as "scaffold tokens").

During BPE training, the algorithm iteratively merges the most frequent adjacent token pairs. Many tokens created during this process are highly frequent at the moment of their creation but are subsequently subsumed by further merges. Consequently, these tokens appear infrequently in the final tokenized corpus, yet they remain in the vocabulary. The authors identify three primary issues caused by these residues:

Token Waste: These tokens occupy capacity in token embeddings and decoding layers without being effectively utilized, leading to wasted parameters and FLOPs (estimated at 1–3% for models ≤8B parameters).
Performance Degradation: Because these tokens are rare in the final corpus, they are under-trained. Models encounter higher error rates when processing inputs that trigger these tokens, particularly in atypical or out-of-distribution scenarios.
Vulnerability: The low frequency and association with noisy text patterns make these tokens a vector for adversarial attacks and reduced robustness against misspellings or jailbreak-style perturbations.

2. Methodology: LiteToken

The authors propose LiteToken, a lightweight, plug-and-play algorithm designed to identify and remove intermediate merge residues from an existing BPE tokenizer without retraining the tokenizer or the model. The process involves three main stages:

A. Identification Algorithm

The method uses a corpus-driven statistical analysis to distinguish true residues from linguistically meaningful low-frequency tokens (such as roots, prefixes, or suffixes). It employs two metrics:

Final/Intermediate (FI) Ratio: This measures the frequency of a token in the final corpus ( $F_1$ ) relative to its total occurrences during the merge process ( $F_1 + F_2$ ). A low ratio indicates the token was mostly merged into larger units.
$R = \frac{F_1}{F_1 + F_2}$
Neighbor Entropy Filtering: To avoid removing valid morphological units (e.g., "re" in "rewrite" or "import" as a standalone word), the algorithm calculates the entropy of the left and right neighboring tokens. Intermediate residues typically have fixed, low-entropy contexts (e.g., "includ" is almost always followed by "e", "ing", etc.), whereas valid roots or standalone words exhibit higher neighbor diversity.
$S(t) = \min \{S_{left}(t), S_{right}(t)\}$
Tokens are classified as residues if their FI ratio is below a threshold ( $r$ ) and their entropy score is below a threshold ( $s$ ).

B. Removal and Re-merging

Once residues are identified, the tokenizer is modified via a Split and Re-Merge process:

Split: When a token belonging to the residue set is encountered, the algorithm recursively reverses the merge operations that created it, splitting it back into its predecessor tokens until only non-residue (or basic) tokens remain. This ensures no unknown tokens are generated.
Re-Merge: To prevent excessive sequence fragmentation, the algorithm applies a re-merge step using the "unmasked" learned merges. This recombines the split tokens into more compact, linguistically aligned units (e.g., re-forming "corruption" from split parts) without reintroducing the removed residue tokens.

C. Output Layer Masking

To prevent the model from generating residue tokens during autoregressive decoding, the corresponding columns in the output projection matrix ( $O \in \mathbb{R}^{d_h \times N_{vocab}}$ ) are masked. The authors note that re-merging is not applied during generation to maintain compatibility with KV caches.

D. Compatibility with Tiktoken

The method is adapted for tiktoken (used by Llama-3, GPT, and Gemma families), which lacks a strict merge tree. For these tokenizers, the algorithm considers all permitted merge paths that yield a specific token and performs frequency statistics based on the actual merge table used during encoding.

3. Key Contributions

Systematic Characterization: The paper provides an empirical analysis revealing that approximately 5% to 10% of tokens in widely used LLM tokenizers (including Qwen, DeepSeek, Llama-3, and GPT families) are intermediate merge residues.
LiteToken Algorithm: A novel, training-free method to prune these residues, reducing vocabulary size and model parameters while improving token efficiency.
Plug-and-Play Deployment: The approach allows pretrained models to utilize the modified tokenizer immediately without fine-tuning, preserving overall performance.

4. Experimental Results

The authors evaluated LiteToken on models ranging from 0.6B to 70B parameters (Qwen, Llama-3, DeepSeek, etc.) using the C4-en, RedPajama, and SQuAD datasets.

Parameter and Compute Reduction: Removing residues reduces vocabulary size by 5–10%, leading to a proportional reduction in embedding and output layer parameters (up to ~2.6% for 0.6B models).
Language Modeling Performance: The modified tokenizer ("Lite") showed negligible impact on perplexity (PPL). For example, on C4-en, the PPL difference between original and Lite tokenizers was less than 0.5 for Qwen-3-8B and Llama-3.1-70B.
Robustness: The most significant gains were observed in robustness. Under perturbation tests (misspellings and adversarial token boundaries), LiteToken models demonstrated higher accuracy in error correction and question answering compared to original models. For instance, the performance drop for Qwen-3-8B under perturbation was reduced from 1.78% to 0.82%.
General Performance: On standard benchmarks like SQuAD, the removal of residues maintained F1 scores within a negligible margin (e.g., 0.833 vs. 0.829 for Qwen-3-8B), confirming that the pruning does not harm general NLP capabilities.

5. Significance and Claims

The paper argues that tokenizer design is a critical, yet often overlooked, component of LLM architecture. The authors claim that intermediate merge residues represent a tangible inefficiency that wastes computational resources and creates vulnerabilities to noisy inputs.

By introducing LiteToken, the authors demonstrate that:

A significant portion of the vocabulary in modern BPE tokenizers is redundant.
Removing these residues improves model robustness against typographical errors and adversarial perturbations without the cost of retraining.
The "plug-and-play" nature of the method allows for immediate deployment benefits in terms of memory savings and inference efficiency.

The authors conclude that while their current work focuses on English and Latin scripts, the phenomenon of intermediate residues is likely prevalent in other languages, suggesting a need for further investigation into tokenizer optimization across diverse linguistic structures. They emphasize that their work is modest in scope, focusing on the tokenization layer, and leaves interactions with model retraining as a direction for future research.