💬 NLP

Compute Optimal Tokenization

This paper demonstrates that in compute-optimal language model configurations, parameter counts scale proportionally to data size measured in bytes rather than tokens, revealing that the optimal token compression rate differs from standard BPE and decreases as compute increases.

Original authors: Tomasz Limisiewicz, Artidoro Pagnoni, Srini Iyer, Mike Lewis, Sachin Mehta, Alisa Liu, Margaret Li, Gargi Ghosh, Luke Zettlemoyer

Published 2026-05-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tomasz Limisiewicz, Artidoro Pagnoni, Srini Iyer, Mike Lewis, Sachin Mehta, Alisa Liu, Margaret Li, Gargi Ghosh, Luke Zettlemoyer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building the ultimate library of knowledge. You have a fixed amount of money (your compute budget) to build this library. You have two main choices:

Buy a massive building with millions of empty shelves (Model Size).
Buy a huge pile of books to fill those shelves (Training Data).

For years, experts told librarians: "To get the best library, buy 20 books for every single shelf you build." But this advice had a hidden flaw: it assumed every book was the same size. In reality, some books are thick encyclopedias, while others are thin pamphlets.

This paper, titled "Compute Optimal Tokenization," argues that we've been measuring our books in the wrong unit. Instead of counting "books" (tokens), we should be counting "pages" (bytes).

Here is the breakdown of their findings using simple analogies:

1. The "Book Size" Problem (Compression Rate)

In the world of AI, text is chopped up into chunks called tokens.

Low Compression: Imagine chopping a sentence into individual letters. You get a huge pile of tiny scraps (many tokens).
High Compression: Imagine grouping whole words or even phrases into single chunks. You get a smaller pile of big chunks (fewer tokens).

The paper asks: Does it matter how big our chunks are?
The answer is yes. The size of the chunk (called the compression rate) changes how efficiently we can build our library.

2. Finding the Sweet Spot: The "Bytes vs. Tokens" Rule

The researchers trained nearly 1,000 different AI models, tweaking the size of the chunks and the size of the model. They discovered a new rule for the perfect balance:

The Old Rule: "Buy 20 tokens per parameter." (This only works if your tokens are a specific size, like standard BPE tokens).
The New Rule: "Buy 60 bytes of text per parameter."

The Analogy:
Think of your AI model as a chef and the data as ingredients.

If you give the chef tiny, pre-chopped ingredients (high compression), they can cook a lot of meals quickly.
If you give them whole vegetables (low compression), they have to chop them first, which takes time.
The paper found that no matter how you chop the vegetables, the chef performs best when they have a specific weight of ingredients (60 bytes) for every unit of their skill (parameter). It doesn't matter if that weight is 20 big chunks or 100 tiny crumbs; the total weight is what matters.

3. The "Goldilocks" Compression Rate

You might think, "If I make the chunks bigger, I save more time, so I should make them as big as possible!"
The paper says: Not so fast.

They found that there is a Goldilocks zone for chunk size.

Too small: The model gets overwhelmed by too many tiny pieces of data.
Too big: The model loses too much detail because the chunks are too coarse.
Just right: There is a specific compression rate that yields the smartest model for a given budget.

The Twist: As you get more money (more computing power), the "Just Right" size actually gets smaller.

Analogy: If you have a small kitchen, you might want pre-chopped veggies to save time. But if you have a massive, professional kitchen with unlimited staff, you might prefer whole vegetables because you can process them so efficiently that the extra detail is worth the effort.

4. One Size Does Not Fit All (Language Matters)

The researchers tested this on different languages (English, Hindi, Arabic, Russian, etc.). They found that the "Just Right" chunk size depends on the language.

The Analogy: Imagine packing suitcases for different trips.
- For a trip to a country where people speak a language with very compact words (like English), you might pack slightly larger items.
- For a trip to a country where words are longer or use different scripts (like Hindi or Arabic), the "optimal" way to pack changes.
The paper found that popular AI tools (like Llama 3 or Qwen) often use a "one-size-fits-all" packing method. This works okay for English but is often too compressed for some languages and not compressed enough for others. They are essentially packing the wrong size items for the specific trip.

Summary of Key Takeaways

Stop counting tokens, start counting bytes. When planning how much data to feed an AI, measure the raw size of the text (bytes), not the number of chunks (tokens). The ratio should be roughly 60 bytes of text for every 1 million parameters in the model.
There is a perfect chunk size. Making tokens too big or too small hurts performance. There is a specific "sweet spot" for compression.
The sweet spot changes. As you get more powerful computers, you should actually use slightly smaller chunks (lower compression) to get the best results.
Language matters. The perfect chunk size is different for Hindi than it is for English. Current AI models often use a generic setting that isn't optimal for every language.

In short: To build the smartest AI possible, don't just follow the old rule of "20 tokens per parameter." Instead, match the weight of the text to the size of the brain, and adjust the size of the text chunks based on how much computing power you have and which language you are teaching it.

Technical Summary: Compute Optimal Tokenization

Problem Statement

Scaling laws have traditionally guided the efficient design of large language models (LLMs) by prescribing the optimal balance between model size and training data volume. However, standard approaches express data volume in tokens, a unit that inherently depends on the specific tokenizer and its compression rate (the average number of bytes per token). This dependency overlooks the information density of the data unit itself. Consequently, existing scaling findings are tied to specific tokenization schemes (e.g., BPE), potentially obscuring the true relationship between compute, data, and model parameters. The paper addresses the gap in understanding how the compression rate ( $T$ ) impacts compute-optimal scaling trends and whether the optimal data-to-parameter ratio should be expressed in tokens or bytes.

Methodology

To systematically investigate the role of compression rate without confounding variables like vocabulary size, the authors employed the Byte Latent Transformer (BLT) architecture. BLT allows for precise control over the compression rate by segmenting byte-level input into latent tokens based on an entropy threshold, enabling the training of models with varying $T$ while keeping the vocabulary size (and thus parameter count) constant.

Experimental Setup:

Models: The study trained 988 latent tokenized models (BLT) and 320 subword tokenized models (using Character, BPE, SuperBPE, and masked BPE variants).
Scale: Models ranged from 50M to 7B parameters.
Compute Budgets: Experiments spanned budgets from $5 \times 10^{18}$ to $2 \times 10^{21}$ FLOPs.
Compression Rates ( $T$ ):
- Latent Models: $T \in \{1, 2, 4, 6, 8, 12\}$ bytes/token.
- Subword Models: $T \in \{1.01, 3.71, 4.16, 4.57, 6.16\}$ bytes/token.
Data: Training was conducted primarily on the DCLM corpus (plain English), with extensions to five other languages (French, Vietnamese, Russian, Arabic, Hindi) and an artificially inflated English dataset.
Evaluation: Models were evaluated using Bits-Per-Byte (BPB) on the C4 validation split, ensuring context length was measured in bytes (e.g., 8192 bytes) rather than tokens to maintain consistency across compression rates.

Analysis Approach:
The authors fitted power laws in two stages:

Scaling Law I: Determined the optimal training data size ( $B^\star$ ) and model size ( $N^\star$ ) for a given compute budget ( $C$ ) and compression rate ( $T$ ) by finding the minimum of IsoFLOP curves (loss vs. data/parameters).
Scaling Law II: Modeled the dynamics of the optimal loss ( $L^\star$ ) as a function of $C$ and $T$ , specifically investigating the existence of an optimal compression rate.

Key Contributions and Results

1. Scaling Law I: Bytes vs. Tokens

The study establishes that in compute-optimal configurations, the optimal ratio between training data bytes and model parameters remains approximately constant, regardless of the compression rate.

Finding: The optimal bytes-per-parameter ratio ( $\rho^\star$ ) is close to 60 bytes per parameter for English data.
Implication: The widely cited "Chinchilla" rule of ~20 tokens per parameter is an artifact of the specific compression rate of BPE tokenizers ( $T \approx 4.57$ ). When generalizing scaling recipes across different tokenizers, developers should match the ratio of training bytes to parameters, not tokens to parameters.
Mathematical Form: $B^\star(C, T) \approx B_0 C^\alpha T^\beta$ and $N^\star(C, T) \approx N_0 C^{1-\alpha} T^{1-\beta}$ , where $\alpha \approx 0.465$ and $\beta \approx 0.471$ . Since $\alpha$ and $\beta$ are close to 0.5, the ratio $\rho^\star = B^\star/N^\star$ remains nearly constant across varying $C$ and $T$ .

2. Scaling Law II: Optimal Compression Rate

The paper discovers that there exists a specific optimal compression rate ( $T^\star$ ) that minimizes loss for a fixed compute budget.

Non-Monotonicity: Loss is non-monotonic with respect to compression; deviating from $T^\star$ in either direction (too low or too high compression) increases loss.
Compute Dependence: The optimal compression rate decreases as the compute budget increases. For example, at $C=10^{20}$ FLOPs, $T^\star \approx 3.69$ , while at $C=2 \times 10^{21}$ FLOPs, $T^\star \approx 3.33$ .
Subword Validation: This trend holds for subword tokenized models as well. Surprisingly, models with 90% of the BPE vocabulary masked (lower effective compression) outperformed standard BPE models at large scales, suggesting that lower compression may be beneficial for very large models.

3. Multilingual Generalization

The findings extend to languages beyond English, with optimal values correlating with cross-lingual parity (the ratio of byte length required to express the same information in different languages).

Language Specificity: Languages with higher parity (lower information density per byte, e.g., Hindi, Russian) require a higher optimal compression rate and a higher optimal bytes-per-parameter ratio.
Current Tokenizers: Popular multilingual tokenizers (e.g., Llama 3, Qwen 3) tend to over-compress high-resource languages (like English) and under-compress low-resource languages (like Hindi), diverging from the computed optimal $T^\star$ .

4. Inference Trade-offs

While lower compression (closer to the optimal $T^\star$ for training) yields better performance, it increases inference compute cost per byte. The paper notes a trade-off: higher compression reduces inference costs but may degrade performance if it deviates significantly from the training-optimal rate.

Significance and Claims

The paper claims to provide the first thorough study of the impact of compression rate on the compute efficiency of language models. Its primary significance lies in:

Generalizing Scaling Laws: It decouples scaling laws from specific tokenizers by establishing bytes as the fundamental unit for data scaling, rather than tokens.
Optimal Tokenization Strategy: It demonstrates that the "optimal" compression rate is not static but depends on the training compute budget and the specific language domain.
Guidance for Developers: It advises model developers to select tokenization schemes that align with the compute-optimal bytes-per-parameter ratio and to consider language-specific compression rates, particularly for multilingual models where standard subword tokenizers often fail to achieve optimal compression.

The authors conclude that while latent tokenization (BLT) offers precise control to discover these trends, the scaling laws derived hold consistently for subword tokenized models, providing actionable guidelines for the selection of tokenization schemes to maximize compute efficiency.