Equity with Efficiency: An Empirical Study of Tokenizers for Multilingual Large Language Models

Technical Summary: Equity with Efficiency: An Empirical Study of Tokenizers for Multilingual Large Language Models

Problem Statement
Multilingual Large Language Models (LLMs) rely on subword tokenization to convert discrete text into continuous neural representations. However, state-of-the-art models predominantly utilize Byte-level Byte-Pair Encoding (BPE), a strategy that structurally favors high-resource languages and Latin scripts. This bias arises because Latin characters are encoded as single bytes, whereas non-Latin characters often require multiple bytes. Consequently, for speakers of underrepresented languages, particularly in Southeast Asia (SEA), this disparity inflates inference costs, degrades long-context reasoning, and widens cross-lingual capability gaps. While several equitable tokenization methods have been proposed—such as Parity-aware BPE, Morphology-Driven Byte Encoding (MYTE), and Byte Latent Transformer (BLT)—they have not been systematically compared under uniform experimental conditions to determine their relative efficacy in balancing efficiency and equity.

Methodology
This study presents the first systematic comparison of equitable tokenizers across a unified benchmark spanning 11 Southeast Asian languages: English, Burmese, Chinese, Indonesian, Khmer, Lao, Malay, Tagalog, Tamil, Thai, and Vietnamese.

• Tokenizers Evaluated: The study compares a baseline Byte-level BPE against three alternative approaches:
  • Parity-aware BPE (PA BPE): Optimizes merge frequencies to improve the worst-case compression rate across languages.
  • Morphology-Driven Byte Encoding (MYTE): Replaces character-based byte codes with morpheme-based codes to achieve balanced morphological coverage.
  • Byte Latent Transformer (BLT): Operates directly over dynamic byte patches without a fixed vocabulary, using an entropy model for segmentation.
• Experimental Design: To ensure fairness and reproducibility, all tokenizers were trained on the same dataset (1 million sentences from mC4). Subsequently, 1.5B-parameter decoder-only language models were trained from scratch using these tokenizers on a controlled dataset of 100 million sentences from FineWeb2. Vocabulary sizes were matched where possible (45k, 90k, 135k tokens), and computational costs were normalized to an 8× NVIDIA H200 equivalent.
• Evaluation Metrics:
  • Intrinsic: Tokenizer parity (ratio of tokens per sentence relative to English), Gini coefficient (distribution of tokenization costs), and compression rate.
  • Extrinsic: Downstream performance on English classification (PIQA, HellaSwag, Arc-C), multilingual classification (XNLI, XCOPA, XStoryCloze), and machine translation (BLEU and chrF scores) across the 10 non-English SEA languages.

Key Results

• Efficiency vs. Equity Trade-off: Parity-aware BPE lies on the Pareto frontier of the efficiency-equity space. It achieves the lowest Gini coefficient (indicating high equity) while maintaining competitive compression rates comparable to standard Byte-level BPE.
• Downstream Performance:
  • English Tasks: Standard Byte-level BPE achieved the highest scores on English classification benchmarks, with statistically significant advantages over alternatives in most cases.
  • Multilingual Tasks: Performance was task-dependent. MYTE significantly outperformed other tokenizers on XNLI (natural language inference), leveraging its morphologically richer representations for semantic reasoning. However, Byte-level BPE performed best on XCOPA and XStoryCloze, suggesting its efficiency benefits tasks requiring causal and narrative reasoning.
  • Machine Translation: MYTE achieved the highest BLEU scores in both translation directions (EN→XX and XX→EN), benefiting from finer-grained generation of morphologically complex target forms.
• BLT Performance: Despite its architectural novelty, BLT underperformed on downstream tasks. The authors attribute this to the entropy model's misalignment with the constraints of limited low-resource training data, resulting in poor equity and high token parity.
• Scaling Effects: Increasing vocabulary size improved efficiency for Byte-level BPE but exacerbated inequity for Parity-aware BPE, as English acquired more merges than other languages. MYTE remained relatively insensitive to vocabulary scaling within the tested range.

Significance and Claims
The paper claims that cross-lingual fairness and tokenization efficiency are not fundamentally at odds. The findings offer practical guidance for designing equitable multilingual models:

1. Parity-aware BPE is recommended as a responsible default for multilingual models targeting SEA languages due to its favorable position in the efficiency-equity space and strong downstream performance, provided parallel data is available for training.
2. MYTE is preferred when morphology and translation quality are critical, assuming the computational budget can accommodate its higher training overhead and lower compression efficiency.
3. Tokenizer Choice as a Determinant: The study establishes that tokenizer choice is the primary factor affecting tokenizer parity, explaining the majority of variance in inference costs across languages. This choice directly impacts the economic accessibility of multilingual models for underrepresented language communities, as token-priced APIs impose unequal costs based on script type when using biased tokenizers.

The authors conclude that equitable tokenization has direct, quantifiable consequences for the hundreds of millions of speakers in Southeast Asia, and that addressing tokenization bias is a viable path toward more inclusive AI systems.