💬 NLP

The Token Tax: Systematic Bias in Multilingual Tokenization

This paper demonstrates that tokenization inefficiency imposes a "token tax" on morphologically complex, low-resource African languages by correlating higher token fertility with lower accuracy and significantly increased costs, while highlighting that reasoning models can help narrow these performance gaps.

Original authors: Jessica M. Lundin, Ada Zhang, Nihal Karim, Hamza Louzan, Victor Wei, David Adelani, Cody Carroll

Published 2026-03-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jessica M. Lundin, Ada Zhang, Nihal Karim, Hamza Louzan, Victor Wei, David Adelani, Cody Carroll

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to send a package across the world. For some people, the package is a small, neat box that fits perfectly into a standard shipping container. For others, their "package" is the same size in reality, but because of how it's wrapped, it gets chopped into hundreds of tiny, awkward pieces. To ship it, you need a truck that is four times bigger, it costs four times as much, and it takes four times longer to deliver.

This paper, "The Token Tax," argues that this is exactly what is happening to many languages in the world of Artificial Intelligence (AI).

Here is the breakdown in simple terms:

1. The Problem: The "Wrapping" Issue (Tokenization)

AI models don't read words like humans do; they read them as chunks called tokens. Think of tokens like LEGO bricks.

English is like a language where most words are already single LEGO bricks. The word "cat" is one brick.
Morphologically complex languages (like many African languages, or even German) are like languages where a single word is a whole LEGO castle. To fit it into the AI's system, the AI has to break that castle apart into 5, 10, or even 20 tiny bricks just to represent one word.

The paper calls this "Fertility." High fertility means a language needs many tokens to say one word. Low fertility (like English) needs few tokens.

2. The "Token Tax": Why It Costs More

The authors discovered that this "extra wrapping" isn't just a minor inconvenience; it's a massive financial and performance tax.

The Quadratic Penalty: AI models are like a party where everyone has to talk to everyone else. If you double the number of people (tokens), the amount of talking (computing power) doesn't just double; it quadruples (2 x 2 = 4).
The Cost: If a language needs twice as many tokens to say the same thing as English, training an AI on it costs 4 times more money and takes 4 times longer.
- Analogy: Imagine training an AI on English costs $100. Training that same AI on a language with "double fertility" would cost $400. If the language is even more complex (5x tokens), the cost jumps to $2,500.
The Result: Because it's so expensive, companies are less likely to build good AI for these languages. This creates a "digital divide" where billions of speakers are left behind.

3. The Performance Hit: The "Cluttered Desk"

It's not just about money; it's about how smart the AI gets.

The paper tested 10 different AI models on 16 African languages.
They found a direct link: The more tokens a language needs (higher fertility), the dumber the AI gets at answering questions.
Analogy: Imagine trying to read a book where every single word is written out as a long, confusing sentence. You get tired, you lose your place, and you make mistakes. The AI gets the same "tired" feeling when it has to process too many tokens for simple words.

4. The Good News: "Reasoning" Models are Helping

The paper tested a new generation of AI models (like DeepSeek and o1) that are designed to "think" before they answer, rather than just guessing the next word.

The Result: These "reasoning" models are much better at handling these messy, high-token languages. They narrowed the gap between English speakers and African language speakers significantly.
The Catch: They didn't fix the problem completely. The "Token Tax" is still there; the reasoning models just have better tools to pay it.

5. The Big Picture: Why This Matters

The authors argue that we need to stop treating this as a minor technical glitch. It is a systemic inequality.

Economic: It costs companies too much to support these languages, so they won't do it.
Technical: The current AI architecture is fundamentally biased against languages that don't look like English.

The Solution?
We need to invent better "LEGO bricks" (tokenizers) that understand the structure of complex languages so they don't have to be broken into tiny pieces. We also need to change how we price AI services so that speaking a complex language doesn't cost you 4x more.

In short: Right now, the AI world is built for English. If you speak a language that requires more "digital wrapping," you are paying a heavy tax in money, time, and intelligence. This paper is a call to fix the wrapping so everyone gets a fair deal.

1. Problem Statement

The paper addresses the systematic computational and economic inequality inherent in current Large Language Model (LLM) architectures regarding low-resource, morphologically complex languages (specifically African languages).

Tokenization Inefficiency: Morphologically complex languages often require significantly more tokens to represent the same semantic content compared to high-resource languages like English. This metric is termed fertility (tokens per word).
The "Token Tax": Due to the quadratic $O(n^2)$ scaling of attention mechanisms in Transformers, an increase in token count does not result in a linear increase in cost. Instead, a 2× increase in fertility leads to a 4× increase in training time, compute cost, energy consumption, and inference latency.
Performance Degradation: This inefficiency creates a "token tax" that disproportionately penalizes speakers of these languages, leading to lower model accuracy and higher barriers to entry for training or fine-tuning models in these languages.

2. Methodology

The authors conducted a large-scale empirical study to quantify the relationship between tokenization efficiency and model performance.

Dataset: The study utilized AfriMMLU, a benchmark containing 9,000 multiple-choice question and answer (MCQA) items across 16 African languages and 5 subjects (Elementary Mathematics, Global Facts, High School Geography, High School Macroeconomics, and International Law).
Models Evaluated: The researchers evaluated 10 distinct LLMs, including both non-reasoning models (e.g., Llama 3.1 405B, GPT-4o, Qwen-2.5) and reasoning models (e.g., DeepSeek R1/V3, OpenAI o1).
Metrics:
- Fertility: Calculated as the ratio of tokens to words ( $F = T/W$ ) for each language using specific model tokenizers.
- Accuracy: Measured via MCQA inference on the AfriMMLU dataset.
- Statistical Analysis: Linear regression was performed to model the relationship between fertility (independent variable) and accuracy (dependent variable) for each model-subject pair. Mixed-effects models were used to account for language-specific variations.
Economic Modeling: The authors extrapolated token counts to estimate training costs (in USD and petaFLOP-days) and inference costs/latency, applying the quadratic scaling law of Transformers.

3. Key Contributions

Fertility as a Predictor: The study confirms that fertility is a reliable predictor of MCQA accuracy. Higher token counts per word consistently correlate with lower model performance across all evaluated models and subjects.
Reasoning Models Analysis: This is the first large-scale comparison of tokenization effects on reasoning vs. non-reasoning LLMs. The authors found that while reasoning models (like DeepSeek and o1) significantly narrow the accuracy gap between high-resource and low-resource languages, they do not eliminate the bias rooted in tokenization inefficiency.
Economic Quantification: The paper translates technical token inflation into concrete economic terms, demonstrating the "token tax" as a barrier to equitable NLP development.
Data Release: The authors released public datasets containing model results from the AfriMMLU benchmark (including reasoning models) and MMLU token metrics.

4. Key Results

Accuracy vs. Fertility Correlation:
- Across all 10 models and 5 subjects, higher fertility consistently predicted lower accuracy.
- Regression Slopes: Ranged from -0.08 to -0.18. This implies that for every additional token per word, accuracy drops by 8 to 18 percentage points, depending on the model and subject.
- Variance Explained: Fertility explained 20% to 50% of the variance in accuracy ( $R^2$ ), with some specific combinations (e.g., Llama-3.1-405B on Microeconomics) showing highly significant negative correlations ( $p < 0.002$ ).
Performance Gaps:
- African languages trailed English by an average of 25 accuracy points in baseline non-reasoning models.
- Reasoning Models: Models like DeepSeek and o1 reduced this gap by 8–12 points on average. In the "Global Facts" category, the gap narrowed from 25 points to 12–14 points. However, a significant disparity remained.
Economic Impact (The Token Tax):
- Training Costs: Doubling the token count (2× fertility) results in a 4× increase in training cost and time due to $O(n^2)$ $O (n^{2})$ scaling.
  - Example: Training Llama-3.1-405B costs $105M in English but $420M in a language with double the fertility.
- Inference Costs: Generating 1M English-equivalent tokens costs $5–20 in English but $10–40 in a 2× fertility language.
- Latency: Inference latency doubles (e.g., a 2-second prompt in English takes ~4 seconds in a high-fertility language).

5. Significance and Conclusion

The paper argues that tokenization bias is not a minor technical artifact but a systemic barrier to equitable Natural Language Processing (NLP).

Structural Disadvantage: The current architecture creates a compound disadvantage where low-resource languages face higher costs, longer training times, and lower performance simultaneously.
Limitations of Current Solutions: While advanced reasoning capabilities improve performance, they cannot fully compensate for the fundamental inefficiencies introduced by suboptimal tokenization.
Call to Action: The authors advocate for a multi-faceted approach to solve this issue:
1. Technical: Development of morphologically aware tokenization and efficient attention mechanisms.
2. Economic: Implementation of fair pricing structures that do not penalize high-fertility languages.
3. Benchmarking: Expansion of multilingual evaluation datasets (like AfriMMLU) to monitor and drive progress in equitable NLP.

In summary, the paper demonstrates that without addressing tokenization inefficiencies, the NLP field risks excluding billions of speakers from the benefits of language technology, both economically and functionally.