💬 NLP

Calibrating Beyond English: Language Diversity for Better Quantized Multilingual LLM

This paper demonstrates that replacing English-only calibration sets with diverse, language-specific, or multilingual data significantly improves the perplexity of quantized multilingual LLMs, revealing that linguistic alignment is crucial for robust performance across different languages and quantization methods.

Original authors: Everlyn Asiko Chimoto, Mostafa Elhoushi, Bruce A. Bassett

Published 2026-01-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Everlyn Asiko Chimoto, Mostafa Elhoushi, Bruce A. Bassett

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, incredibly smart library (a Large Language Model) that knows how to speak dozens of languages. However, this library is so huge that it takes up an entire warehouse of space and requires a giant, expensive generator to run. To make it fit in a small shed and run on a standard power outlet, you need to "compress" it. This process is called quantization.

Think of quantization like packing a suitcase for a trip. You have to squeeze everything in, which means you might have to fold clothes tightly or leave some things behind. If you pack poorly, your clothes come out wrinkled, and you might forget your toothbrush. In the world of AI, "wrinkled clothes" mean the model makes more mistakes, and "forgetting the toothbrush" means it loses the ability to speak certain languages well.

The Problem: The "English-Only" Packing List

For a long time, when people packed these AI suitcases, they used a specific packing list (called a "calibration set") written entirely in English. They assumed that if the suitcase was packed well for English, it would be fine for French, Swahili, or Chinese too.

The authors of this paper asked: "Is it fair to pack a multilingual suitcase using only an English checklist?"

They tested this by packing the same AI model (Llama3.1 and Qwen2.5) using different packing lists:

English-only: The old standard.
Single-language: A list in just French, or just Swahili, etc.
Multilingual: A mixed bag containing a little bit of everything.

The Big Discovery

The results were clear: The English-only packing list was the worst choice for a multilingual suitcase.

The "One-Size-Fits-All" Myth: Using an English list to pack a multilingual model is like trying to fit a pair of boots, a pair of sandals, and a winter coat into a box designed only for boots. It doesn't fit right. The model performed worse in non-English languages when packed with English data.
The Multilingual Mix Wins: When they used a multilingual packing list (a mix of many languages), the model came out of the suitcase much less wrinkled. It spoke all languages better, with fewer mistakes. In some cases, this improved the model's "perplexity" (a score for how confused the model is) by a significant amount—up to 3.5 points lower, which is a huge deal in AI.

Why Does This Happen? (The "Tail" Analogy)

The paper explains this using the concept of "activation tails."

Imagine the AI model is a musician playing a piano.

English data is like practicing only on the middle keys of the piano.
Multilingual data forces the musician to practice the very low bass notes and the very high, squeaky high notes too.

When you compress the model, you have to decide how loud or soft the notes can get. If you only practiced on the middle keys (English), you might set the volume limits too low. When the model tries to play a rare, loud note in Swahili or Chinese later, it gets "clipped" (cut off), and the sound is ruined.

By using a multilingual calibration set, the model sees a wider range of "notes" (rare words, strange symbols, different sentence structures) during the packing process. This teaches the model how to handle the extreme highs and lows of different languages without breaking.

It Depends on the "Packing Method"

The paper also found that different compression tools react differently to the packing list:

GPTQ (The Sensitive Tool): This method is like a very precise, delicate folding machine. It is extremely sensitive to what's in the suitcase. If you give it an English list, it folds everything for English and messes up the rest. But if you give it a balanced multilingual list, it folds everything perfectly. It needs a diverse mix to work well.
AWQ (The Flexible Tool): This method is a bit more flexible. It can handle an English list okay, but it works best if you give it a list that matches the specific language you want to use. However, if you need the model to speak many languages, a multilingual list is still the safest bet.

The Takeaway

The authors conclude that we need to stop using a "one-size-fits-all" approach.

If you want a model that speaks many languages well, do not just use English data to prepare it.
You need a diverse, multilingual training list to ensure the model is ready for the weird, rare, and extreme parts of every language it might encounter.

In short: To pack a multilingual AI suitcase, you need a multilingual checklist. Using only English leaves the other languages wrinkled and confused.

Technical Summary: Calibrating Beyond English

Problem Statement
Quantization is a standard technique for reducing the storage and computational costs of Large Language Models (LLMs), enabling deployment on resource-constrained infrastructure. However, post-training quantization (PTQ) often degrades model performance. Current best practices overwhelmingly rely on English-only calibration sets (e.g., C4, Pile) to determine quantization parameters, even for multilingual models. This paper investigates whether this "one-size-fits-all" English calibration strategy limits performance across diverse languages, particularly for low-resource and non-Latin script languages where quantization degradation is known to be most severe.

Methodology
The authors conducted a systematic evaluation of eight distinct calibration settings across two quantization algorithms (GPTQ and AWQ) and two multilingual LLMs (Llama3.1 8B and Qwen2.5 7B). The study controlled for total token budget to isolate the effect of language composition.

Calibration Sets: The study compared:
1. English Baselines: Standard English sets (C4 for GPTQ, Pile for AWQ).
2. Single-Language Translated: English sets translated into French, Swahili, Chinese, and isiXhosa.
3. Single-Language Native: Random samples from C4 and Wikipedia in the target languages.
4. Multilingual Mixes: Variants including unbalanced mixes, equal allocation from 10 languages (multi10), and uniform sampling from 112 languages.
5. Code/Math Augmentation: Sets augmented with mathematical and code data to test the impact of rare/structured tokens.
Evaluation: Performance was measured via perplexity on Wikipedia and C4 validation sets across 10 languages (English, French, Swahili, Chinese, isiXhosa, Sotho, Zulu, Yoruba, Igbo, Hausa). Downstream task accuracy (XNLI, XStoryCloze, Global MMLU) and internal model statistics (activation distributions, Hessian distances, quantization error) were also analyzed.

Key Contributions

Systematic Language Analysis: This is the first systematic study isolating language composition as the primary variable in quantization calibration, demonstrating that multilingual calibration is a primary control factor for stability, not a secondary consideration.
Algorithm-Specific Insights: The paper distinguishes how different quantizers interact with calibration data. It identifies that GPTQ (Hessian-based) is highly sensitive to calibration distribution, while AWQ (activation-aware scaling) is more robust but benefits from language alignment.
Mechanism Explanation: Through activation distribution analysis, the authors explain why multilingual sets work: they exhibit longer activation tails and broader vocabulary coverage, capturing outlier tokens and extreme values that single-language sets miss, thereby reducing quantization clipping errors.

Results

Perplexity Gains: Non-English and multilingual calibration sets consistently outperformed English-only baselines. Multilingual mixes achieved the largest overall reductions, with up to 3.52 perplexity points improvement on Llama3.1 8B using GPTQ.
Language Alignment: Tailoring calibration sets to the specific evaluation language yielded the largest improvements for individual languages (e.g., Swahili calibration improved Swahili benchmarks). Furthermore, calibration in one language (e.g., isiXhosa) improved performance in closely related languages (Sotho, Zulu).
Quantizer Differences:
- GPTQ: Showed high sensitivity to calibration shifts (large $\Delta$ PPL swings). It performed best with balanced multilingual mixes (specifically multi10), which stabilized the second-order statistics (Hessian) used for error compensation.
- AWQ: Showed modest variation across calibration sets. It benefited most from language-matched calibration for specific targets but remained near-optimal with multilingual sets for cross-language robustness.
Downstream Tasks: Improvements in perplexity translated to downstream task accuracy. Multilingual calibration consistently outperformed English-only baselines on XNLI, XStoryCloze, and Global MMLU.
Failure Cases: The study identified specific failure cases (e.g., French calibration for AWQ on Llama) where the calibration activation range was narrower than the test distribution, leading to conservative quantization thresholds and performance degradation.

Significance and Claims
The paper argues that static, English-only calibration is suboptimal for multilingual LLMs. The authors claim that linguistic diversity in calibration data is crucial for robust quantization, particularly for low-resource languages. By capturing a broader range of activation distributions and token contexts, multilingual calibration reduces quantization error and preserves model accuracy.

The authors propose three practical guidelines based on their findings:

General Deployment: Use a single, balanced multilingual calibration set (e.g., multi10) for general multilingual systems.
AWQ Optimization: If optimizing for a single language, use a language-matched calibration set, with multilingual sets as a fallback for robustness.
GPTQ Optimization: Prefer balanced multilingual calibration; adding code/math tokens can further stabilize the second-order statistics required by GPTQ.

The paper concludes that while perplexity is the primary metric used, the observed trends suggest that calibration design is a pivotal factor in maintaining multilingual LLM accuracy under low-precision constraints, especially in GPU-poor environments.