English K_Quantization of LLMs Does Not Disproportionately Diminish Multilingual Performance

Technical Summary: English K-Quantization of LLMs Does Not Disproportionately Diminish Multilingual Performance

Problem Statement
The deployment of Large Language Models (LLMs) on consumer-grade hardware is increasingly reliant on quantization techniques, specifically the GGUF format and $k$-quantization. These methods reduce model size by assigning fewer bits to weights deemed less important, a determination made via an "importance matrix"—a text document representing standard use cases. For the vast majority of publicly available quantized models, this importance matrix is written in English.

This creates a critical open question: Does the reliance on English-centric importance matrices sacrifice multilingual performance to preserve English performance? Furthermore, could generating importance matrices in non-English languages (e.g., Norwegian or Malayalam) improve performance in those specific languages? While previous studies suggest quantization impacts multilingual tasks, the specific interaction between the language of the importance matrix and the resulting multilingual capabilities of the model remains under-investigated.

Methodology
The authors conducted a controlled experiment using the Llama 3.3 70B model, selected for its demonstrated proficiency in Norwegian despite not being explicitly trained for it. The study involved the following steps:

1. Quantization Variables: The model was quantized using $k$-quantization (specifically $Q4\_K\_S$, $Q3\_K\_S$, and $Q2\_K\_S$ levels) based on three distinct importance matrices:
   • English: The standard baseline.
   • Norwegian: Machine-translated from the English source text (with untranslatable riddles removed).
   • Malayalam: Machine-translated from the English source text.
2. Evaluation Dataset: Performance was evaluated using the MixEval dataset (version 2024-06-01), chosen for its high correlation with the Chatbot Arena leaderboard and its dynamic construction to avoid training data contamination.
3. Test Conditions:
   • Languages: Evaluation was performed on both English and Norwegian subsets of MixEval.
   • Tasks: The study analyzed both multiple-choice questions (MCQ) and free-form generation tasks.
   • Parameters: Generation temperature was set to 0 for deterministic output. For MCQs, the model was constrained to output a valid choice index within the first token to ensure objective evaluation.
4. Statistical Analysis: Results were compared against a full-precision (FP16) baseline. Statistical significance was determined using Monte Carlo simulations to calculate p-values, assuming the English quantized model's success probability modeled the underlying binomial distribution. A Bonferroni correction was applied to the significance threshold ($\alpha$) to account for multiple hypothesis testing.

Key Results
The experiments yielded the following findings:

• No Disproportionate Harm: Quantizing the model with an English importance matrix did not disproportionately harm performance on Norwegian tasks compared to English tasks. The performance drop observed in Norwegian was consistent with the general degradation seen in English tasks as quantization levels increased (e.g., moving from FP16 to $Q2\_K\_S$).
• No Benefit from Non-English Matrices: Using importance matrices written in Norwegian or Malayalam did not result in statistically significant improvements in performance on Norwegian or English test sets, respectively.
  • In some instances, models quantized with Norwegian matrices performed slightly worse on Norwegian datasets than those quantized with English matrices, though this variance was attributed to random noise rather than a systematic pattern.
• Statistical Significance: All calculated p-values for the comparisons between language-specific quantizations and the English baseline were non-significant (the lowest p-value was 0.03428, which remained above the corrected significance threshold of 0.00417).
• Compression vs. Accuracy: While quantization significantly reduced file sizes (up to ~5.8x compression for $Q2\_K\_S$), the accuracy loss was relatively low. However, accuracy on the Norwegian dataset was consistently lower than on the English dataset across all quantization levels, likely due to the model's inherent lower proficiency in Norwegian and difficulties in translating technical terms.

Key Contributions

1. Empirical Validation of Current Practices: The study provides empirical evidence that the standard practice of using English importance matrices for $k$-quantization does not unfairly penalize multilingual capabilities.
2. Investigation of Importance Matrices: It challenges the hypothesis that tailoring the importance matrix to a specific target language (via translation) yields performance gains for that language.
3. Benchmarking Framework: The paper demonstrates a rigorous methodology for evaluating quantization effects on multilingual LLMs using the MixEval dataset, separating the effects of quantization from the inherent language capabilities of the base model.

Significance and Claims
The paper concludes that current quantization practices, specifically the use of English-centric importance matrices, are safe for multilingual deployment. The authors assert that there is no evidence to suggest that English quantization "sacrifices" multilingual performance. Conversely, they claim that generating importance matrices in non-English languages does not offer a statistically significant advantage and may even introduce minor, non-significant performance degradation.

The authors maintain a modest tone, acknowledging that their study covers only a specific combination of model (Llama 3.3 70B), languages (English, Norwegian, Malayalam), and quantization schemes ($k$-quantization). They note that the inability to perform human evaluation of "naturalness" in speech is a limitation, as previous work suggests quantization may affect the fluency of multilingual output even if accuracy metrics remain stable. Ultimately, the paper suggests that for consumer-grade local deployment, the convenience of English-based quantization does not come at the cost of multilingual utility.