💬 NLP

The Tokenizer Tax Across 25 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penalty

This paper quantifies a "tokenizer tax" across 25 European languages, revealing that non-English languages, particularly Ukrainian and those with complex morphology, suffer from significantly higher token-to-word ratios due to underrepresentation in pre-training data, while demonstrating that these fertility rankings remain consistent across domains and that few-shot effects are model-intrinsic rather than language-dependent.

Original authors: Volodymyr Ovcharov

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Volodymyr Ovcharov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are paying for a taxi ride. In English, the taxi charges you by the "stop" (a word). But in Ukrainian, the taxi driver insists on charging you by the "step" (a tiny piece of a word). Because Ukrainian words are often longer and more complex, the driver takes many more steps to get to the same destination. You end up paying twice as much for the exact same ride, even though the distance hasn't changed.

This paper, written by Volodymyr Ovcharov, is all about that hidden "tax" on non-English languages when using AI. Here is the breakdown of what they found, using simple analogies:

1. The "Step" Tax (Tokenizer Fertility)

AI models don't read whole words like humans do. They chop text into tiny chunks called tokens.

The Analogy: Think of a word as a loaf of bread.
- English: The AI cuts the loaf into big, thick slices. You only need 1.2 slices to represent one word.
- Ukrainian: The AI uses a dull knife and cuts the same loaf into tiny, crumbly pieces. It takes 2.7 slices to represent one word.
The Cost: Since AI companies charge you per "slice" (token), speaking Ukrainian costs you roughly 2.2 times more than speaking English for the same amount of text.
The Hierarchy: The paper measured 25 European languages.
- The Cheap Group: English, Spanish, and French (1.2 to 1.7 slices per word).
- The Middle Group: German and Dutch (1.7 to 1.9 slices).
- The Expensive Group: Slavic languages like Polish and Czech (2.2 to 2.5 slices).
- The Most Expensive: Ukrainian, Greek, and Maltese (around 3.1 slices).

2. The "Ukrainian Penalty"

The researchers found something surprising about Ukrainian. Even though Ukrainian, Polish, and Czech are all Slavic cousins with similar word structures, Ukrainian is significantly more expensive to process.

The Analogy: Imagine two identical factories. One (Polish) has a well-stocked supply of pre-cut bricks because it's been building for a long time. The other (Ukrainian) has to cut every brick from raw stone because it was ignored in the past.
The Cause: The paper shows that Ukrainian has 2 to 6 times less training data in the AI's "library" compared to Polish or Czech. Because the AI hasn't seen Ukrainian words as often, it doesn't know how to group them efficiently. It keeps chopping them up into tiny, inefficient pieces.

3. The "One-Size-Fits-All" Knife

The paper tested if this "tax" changes depending on what you are talking about (e.g., legal contracts vs. news vs. Wikipedia).

The Finding: It doesn't matter. The ranking of which AI is "cheap" and which is "expensive" stays exactly the same whether you are talking about football, law, or history.
The Takeaway: If you test an AI on one type of text, you know exactly how much it will cost you for any other type of text. You don't need to re-test it every time.

4. The "Magic Trick" (Few-Shot Learning)

AI can sometimes learn a new task just by seeing a few examples (like showing it a sample question and answer). The researchers tested if this "magic trick" worked differently for different languages.

The Finding: The trick isn't about the language; it's about the AI's personality (its design).
- Some AIs (like "Nemotron") get smarter when shown examples, no matter if the text is Ukrainian, Polish, or Russian.
- Other AIs (like "Maverick") actually get dumber when shown examples, again, regardless of the language.
The Lesson: Don't blame the language for the AI's confusion. Blame the AI's design. If an AI fails with examples in English, it will likely fail with examples in Ukrainian too.

5. Why Some AIs Are Better at Chopping

The researchers looked under the hood to see how different AIs cut the words.

The Good Choppers: Some AIs (like Gemma 2) cut Ukrainian words at natural "morpheme" boundaries (the meaningful parts of a word, like roots and endings). This is efficient.
The Bad Choppers: Other AIs (like Qwen3) cut words at random spots, sometimes splitting a single meaningful part in half. It's like cutting a sandwich right through the middle of a pickle instead of between the slices of bread.
The Surprise: Having a bigger dictionary (vocabulary) doesn't guarantee better cutting. Some AIs with huge dictionaries still chopped Ukrainian words into tiny, inefficient pieces because they just didn't have enough Ukrainian examples in their training data to learn the right cuts.

Summary of Recommendations

The paper suggests three simple rules for anyone using AI with Ukrainian or similar languages:

Expect the Tax: Budget for the fact that Ukrainian will cost about double what English costs.
Test Once: You only need to measure the "tax" once; it applies to all topics.
Be Careful with Examples: Don't assume that showing an AI examples will help. For some models, it might actually hurt performance, and this happens in every language.

The Bottom Line: The AI world is currently built with an English bias. Until the "libraries" of training data are balanced, languages like Ukrainian will pay a hidden "step tax" just to be understood.

Technical Summary: The Tokenizer Tax Across 24 European Languages

Problem Statement

Tokenizers impose a hidden economic and technical cost on non-English Natural Language Processing (NLP), particularly for morphologically rich languages. This "tokenizer tax" arises from fertility—the number of subword tokens generated per word. While English text is efficiently tokenized (approx. 1.2 tokens/word), languages with complex morphology or underrepresentation in pre-training data can incur costs 2.5 times higher. Previous studies (Petrov et al., 2023; Ahia et al., 2023) documented these disparities but relied on heterogeneous text sources, making it difficult to isolate the tokenizer's contribution from content variation. Furthermore, the specific "Ukrainian penalty" relative to cognate Slavic languages and the cross-lingual consistency of few-shot learning effects remained unquantified in a controlled setting.

Methodology

The study employs a controlled cross-lingual experimental design to measure fertility and downstream effects across ten foundation models and 25 European languages.

Models: Ten models were evaluated, including seven accessed via AWS Bedrock (Llama 4 Maverick, Llama 3.3 70B, Mistral Large 3, Nemotron Super 3, Nova Pro, Qwen3 32B, Qwen3 235B) and three measured locally using public tokenizers (GPT-4o, Gemma 2 27B, DeepSeek V3).
Datasets:
- Fertility Measurement: SIB-200 parallel text (25 EU languages + Ukrainian) and Ukrainian Wikipedia (encyclopedic register). Texts were concatenated into ~6,000-character blocks to minimize system prompt overhead.
- Classification: SIB-200 topic classification (7 classes) across four Slavic languages (Ukrainian, Polish, Russian, Czech).
- Linguistic Competence: ULP benchmark (347 multiple-choice questions on Ukrainian grammar).
Experimental Design:
1. Domain Invariance: Measured fertility across legal, news, and encyclopedic registers to test ranking stability.
2. Cross-Lingual Mapping: Calculated mean fertility for 25 languages to establish a hierarchy.
3. Subword Analysis: Decomposed specific Ukrainian legal terms to analyze morphological boundary preservation.
4. Few-Shot Evaluation: Compared zero-shot vs. few-shot accuracy on SIB-200 to determine if performance deltas are language-dependent or model-intrinsic.
5. Competence Correlation: Correlated fertility rates with grammatical accuracy on the ULP benchmark.

Key Contributions

Controlled Cross-Lingual Fertility Map: The first measurement of tokenizer fertility across 25 European languages using parallel text, isolating tokenizer behavior from content variation.
Domain Invariance Validation: Demonstration that fertility rankings are stable across legal, news, and encyclopedic registers ( $\rho > 0.97$ ), implying a single measurement suffices for cost prediction across domains.
Mechanistic Subword Analysis: Identification that high-fertility tokenizers fragment morphological boundaries at arbitrary byte levels rather than preserving morphemes, explaining performance gaps.
Model-Intrinsic Few-Shot Effects: Evidence that few-shot learning effects (improvement or degradation) are consistent across different Slavic languages for a given model, indicating the effect is driven by model architecture rather than input language properties.
The Ukrainian Penalty Quantification: Specific quantification of Ukrainian's 15–18% higher token cost compared to cognate Slavic languages (Polish, Czech), attributed to underrepresentation in pre-training data.

Results

1. The Tokenizer Tax Hierarchy

Fertility spans a 2.5× range across European languages:

Most Efficient: English (1.23 tokens/word).
Romance: 1.5–1.7.
Germanic: 1.7–1.9.
Slavic: 2.2–2.5.
Uralic/Baltic: 2.7–3.0.
Most Expensive: Greek and Maltese (~3.1).

The Ukrainian Penalty: Ukrainian averages 2.66 tokens/word, paying 15–18% more than Polish (2.25) and Czech (2.28). This gap persists despite similar morphological complexity. Corpus analysis reveals Ukrainian has 2–6× less pre-training data than Polish in major corpora (mC4, OSCAR, CulturaX), leading to fewer optimized subword merges.

2. Model Variance and Subword Fragmentation

Model design significantly impacts fertility. For example, on Greek, fertility ranges from 2.29 (Llama 4 Maverick) to 5.73 (Qwen3).

Efficient Tokenizers (e.g., Gemma 2, Llama 3.3): Preserve morpheme boundaries (e.g., splitting вiдповiдальнiсть into 3–4 tokens).
Inefficient Tokenizers (e.g., Qwen3, DeepSeek): Fragment words at byte boundaries (splitting the same word into 8–9 tokens), despite having comparable or larger vocabulary sizes.

3. Domain Invariance

Fertility rankings are perfectly preserved ( $\rho = 1.0$ ) across news, legal, and encyclopedic texts. While absolute token counts vary slightly by domain (encyclopedic text is generally most expensive due to vocabulary diversity), the relative cost ordering of models remains constant.

4. Cross-Lingual Few-Shot Effects

The impact of few-shot prompting is model-intrinsic, not language-dependent:

Nemotron Super 3: Improved performance on all four Slavic languages (avg +10.5 pp).
Llama 4 Maverick: Degrade performance on all four Slavic languages (avg -8.4 pp), with significant class collapse toward dominant categories (e.g., science, politics).
This consistency across Cyrillic and Latin scripts suggests the effect is determined by how a model's attention mechanism processes demonstration content, not by the input language's morphology.

5. Linguistic Competence

The correlation between low fertility and high grammatical accuracy (ULP benchmark) is weak ( $\rho = -0.43$ ) and not statistically significant ( $p=0.34$ ). While Llama 4 Maverick (lowest fertility) achieved the highest accuracy (57.3%), outliers like Nemotron (moderate fertility, lowest accuracy) indicate that architecture and training data composition are stronger predictors of competence than tokenization efficiency alone.

Significance and Claims

The paper claims to provide the first controlled, domain-invariant map of the "tokenizer tax" across Europe. Its primary significance lies in:

Economic Impact: Establishing that processing Ukrainian content costs 2.2× more than English purely due to tokenizer design, a cost that compounds across API calls.
Methodological Shift: Proving that fertility is a stable property of the tokenizer vocabulary, allowing practitioners to measure cost once and predict it across all application domains.
Data Representation: Highlighting that the "Ukrainian penalty" is not solely a linguistic feature but a direct consequence of underrepresentation in pre-training corpora, which results in suboptimal subword vocabularies.
Practical Guidance: Recommending that practitioners prioritize zero-shot approaches for morphologically rich languages (as few-shot effects are unpredictable and model-specific) and budget for the specific token tax of their target language.

The authors release all fertility measurements and subword decomposition data as a public dataset to enable cost-aware model selection and routing policies.