💬 NLP

Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation

This paper demonstrates that widely used normalized metrics for crosslingual language model evaluation introduce biases due to tokenization and orthographic differences, and proposes that sentence-level negative log-likelihood over semantically equivalent sequences offers a more consistent and fair alternative for comparing models across languages.

Original authors: Xiulin Yang, Ethan Gotlieb Wilcox, Catherine Arnett

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiulin Yang, Ethan Gotlieb Wilcox, Catherine Arnett

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, computers are learning to speak and write in dozens of human languages. These digital minds, known as language models, are trained on vast libraries of text to predict what word comes next in a sentence. As these systems grow more powerful, researchers face a critical challenge: how do we know if a model is truly good at a specific language, or if it is just good at the way that language is written down on a computer? To answer this, scientists must measure how well a model understands the meaning of a text, not just how efficiently it can process the raw characters or symbols. The problem is that the tools currently used to measure this understanding often contain hidden traps. They can be skewed by the number of letters in a word, the specific script used to write a language, or the way the computer breaks a sentence into tiny pieces for processing. If these tools are biased, they might make a model look brilliant in one language and clumsy in another, not because of its actual intelligence, but because of technical quirks in how the data is handled. This makes it nearly impossible to fairly compare how well an artificial intelligence performs across the world's diverse tongues.

A team of researchers from Georgetown University and EleutherAI set out to solve this puzzle by testing the very tools used to judge these models. They built a controlled experiment using fifty separate computer models, each trained exclusively on one of ten different languages. To ensure a fair test, they trained these models on parallel data, meaning the models learned from texts that expressed the exact same meaning in different languages. They then tested these models using six different measurement methods that are commonly used in the field. Some of these methods count the total number of tiny building blocks, or tokens, a model uses to understand a sentence. Others count the number of bytes, which are the digital units of storage, or the number of individual characters on the screen. The researchers also tested a method that simply sums up the total difficulty a model has in predicting a whole sentence, without dividing that difficulty by the length of the text.

The results revealed a startling truth about how these measurements work. The methods that divide a model's performance by the number of tokens, bytes, or characters were found to be deeply flawed when comparing different languages. These metrics consistently favored languages that naturally use fewer characters or bytes to express the same idea, such as Chinese, while penalizing languages that require longer sequences of symbols. In one striking example, a model trained only on English was shown to perform "better" on Russian than on English when measured by these flawed methods, simply because the computer broke the Russian text into more pieces, artificially lowering the score. This happened even though the model had never seen Russian before and performed poorly on it in reality. The study showed that these popular metrics were not measuring the model's actual intelligence or understanding; they were merely reflecting the engineering choices made when the computer was set up to read the text.

In contrast, the method that summed up the total difficulty of a sentence without dividing it by length proved to be the only fair approach. This measure, which the researchers call sentence-level negative log-likelihood, looked at the total effort the model exerted to understand the entire meaning of a text. Because it did not try to normalize the score based on how long the text was in terms of letters or digital storage, it remained stable regardless of the language's script or the computer's vocabulary size. When the researchers tested this method on large, real-world multilingual models, it was the only metric that correctly identified which languages the models had learned best, matching the known composition of their training data. The other methods continued to produce rankings that were distorted by the technical details of how the text was stored and processed.

The researchers also investigated whether these measurements remained consistent when the same meaning was expressed in different ways. They found that while some metrics fluctuated wildly depending on which translation or paraphrase was used, the sentence-level method remained steady. This suggests that the apparent stability of the other methods was an illusion created by their sensitivity to surface-level details rather than a sign of true reliability. The study concludes that to truly understand how well artificial intelligence works across different cultures and languages, scientists must stop relying on metrics that count tokens, bytes, or characters. Instead, they must use measures that respect the total meaning of the text, ensuring that the evaluation reflects the model's actual learning rather than the quirks of its digital representation. This shift is essential for building systems that are truly fair and effective for everyone, regardless of the language they speak.

Technical Summary: "Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation"

Problem Statement

Fair evaluation of language models (LMs) across typologically diverse languages remains a fundamental challenge in multilingual Natural Language Processing (NLP). While significant progress has been made in extending language technology to numerous languages, substantial performance gaps persist. A critical barrier to accurately measuring these gaps is the lack of consensus on evaluation metrics. Existing studies employ a variety of downstream tasks and intrinsic metrics (e.g., Perplexity, Bits-per-Byte, Bits-per-Character) with different theoretical justifications. However, empirical investigation into whether these approaches yield meaningful crosslingual conclusions is limited. The central problem is that different metrics often produce substantially different, and sometimes conflicting, rankings of model performance across languages, raising the question of what constitutes a "fair" comparison.

Methodology

The authors propose a systematic empirical investigation to isolate metric behavior from confounds inherent in multilingual models. Their methodology involves three primary experiments:

Controlled Monolingual Evaluation: The authors utilize 50 monolingual language models trained on parallel corpora (PARALLEL-10 and PARALLEL-3) with varying vocabulary sizes (ranging from 8k to 82k tokens) and model scales (GPT2-small and GPT2-medium). This design ensures that observed differences are attributable to the languages and metrics themselves rather than shared parameters or training data imbalances.
Metric Comparison: Six intrinsic metrics are systematically compared:
- Bits-per-Byte (BPB)
- Bits-per-Character (BPC)
- Mean Reciprocal Rank (MRR)
- Perplexity (PPL)
- Token-level Negative Log-Likelihood (Token-NLL)
- Sentence-level Negative Log-Likelihood (Sent-NLL)
Robustness and Generalization Tests:
- Translation Alternatives: Using English-German translation pairs from WMT2019 with multiple reference translations, the authors test metric stability against paraphrasing and translation variations.
- Multilingual Generalization: The findings are validated on nine large multilingual LLMs (including BLOOM, mGPT, XGLM, Llama, Qwen, and Teuken) across four languages (Arabic, English, Russian, Chinese). The authors compare metric rankings against expected rankings derived from training data composition.

Key Contributions and Results

1. Identification of Systematic Biases in Normalized Metrics

The study demonstrates that five of the six widely used metrics introduce systematic biases rooted in engineering choices (tokenization, encoding, and orthography) rather than genuine language modeling quality:

Surface-Level Normalization: Metrics normalized by tokens (PPL, Token-NLL), characters (BPC), or bytes (BPB) are highly sensitive to representation artifacts.
- BPB and BPC: These are biased by orthographic encoding. For instance, languages with logographic scripts (e.g., Chinese) or scripts requiring more UTF-8 bytes per character (e.g., Khmer) receive artificially favorable or unfavorable scores depending on the metric, regardless of model performance.
- PPL and Token-NLL: These are biased by tokenization granularity. Languages with unseen scripts are often over-segmented into smaller units (e.g., bytes), increasing the token count. Since these metrics average likelihood over tokens, a higher token count artificially lowers the score (improving the apparent performance) even if the model has no genuine knowledge of the language.
- MRR: This metric is sensitive to vocabulary size, where smaller vocabularies artificially lower the rank ceiling, making models appear to perform better.
Correlation with Confounds: Spearman's rank correlation analysis reveals strong correlations ( $\rho > 0.8$ ) between these metrics and potential confounds (token count, character count, byte count).

2. Superiority of Sentence-Level NLL (Sent-NLL)

In contrast to normalized metrics, Sentence-level NLL (Sent-NLL)—which sums the negative log-likelihood over a semantically equivalent sequence without normalizing by surface units—satisfies three criteria for a fair metric:

Robustness to Engineering Choices: Sent-NLL shows no significant correlation with token count, character count, or byte count. It is substantially less sensitive to tokenization and encoding choices.
Stability Across Translations: While other metrics yield inconsistent rankings when different translations of the same source text are used, Sent-NLL produces consistent language rankings across translation alternatives.
Generalizability: Sent-NLL rankings align with expected performance based on training data composition in multilingual LLMs. For example, in BLOOM models, Sent-NLL correctly ranks languages according to their data proportions, whereas PPL and Token-NLL are distorted by tokenization artifacts (e.g., ranking Russian highly despite low training data due to over-segmentation).

3. Implications for Downstream Evaluation

The paper discusses challenges in downstream task evaluation, noting that translated benchmarks may introduce noise if translation quality varies, and culture-specific or language-specific tasks (like BLiMP) often lack parallel data or consistent annotation schemas, making crosslingual comparisons difficult.

Significance and Claims

The paper claims to provide the first systematic empirical evidence across multiple metrics, languages, and tokenizer settings demonstrating that normalized metrics are unreliable for crosslingual comparison.

Rejection of Intuition: The authors argue that previous reliance on intuition or theoretical justification for metrics like BPB (claimed to be tokenizer-agnostic) or PPL has been insufficient. Their results show these metrics introduce substantial biases favoring languages with longer sequences or more tokens.
Validation of Prior Work: The study empirically validates the theoretical assumptions underlying prior work that utilizes sentence-level NLL (e.g., Mielke et al., 2019; Cotterell et al., 2018), providing direct evidence for claims that previously lacked it.
Nature of Bias: The authors clarify that while Sent-NLL is not entirely free from confounds, any remaining sensitivity reflects inherent linguistic complexity (e.g., information density) rather than arbitrary engineering choices. They posit that Sent-NLL is the least biased option among commonly used intrinsic metrics, as it normalizes by meaning rather than representation units.
Interpretation of Variability: The paper notes that Sent-NLL rankings show more variability across datasets than normalized metrics. The authors argue this is a strength, reflecting real differences in how well a model has learned each language, whereas the "stability" of other metrics is artificial and driven by engineering artifacts.

The authors conclude that achieving comparable crosslingual evaluation requires moving away from surface-normalized metrics toward approaches that account for semantic equivalence, such as Sent-NLL, to ensure that language rankings reflect modeling quality rather than representation-level factors.