Multilingual Large Language Models do not comprehend all natural languages to equal degrees

1. Problem Statement

Despite the widespread deployment of Multilingual Large Language Models (LLMs), there is a critical lack of understanding regarding their true comprehension capabilities across diverse languages. Current research and benchmarks are heavily skewed toward high-resource, Western, Educated, Industrialised, Rich, and Democratic (WEIRD) languages, particularly English.

The prevailing assumption is that English is the "default" or best-performing language for LLMs, with performance degrading in low-resource or non-WEIRD languages. However, this bias limits the equitable access to reliable AI tools and obscures potential structural limitations in how models process different linguistic systems. The authors argue that existing evaluations often focus on information retrieval or generation rather than the fundamental prerequisite of language comprehension, and they rely on datasets that may not capture the nuances of typologically diverse languages.

2. Methodology

The study employed a rigorous experimental design to evaluate cross-linguistic comprehension and stability across three flagship LLMs and human baselines.

• Models Tested:
  • GPT-4o (OpenAI)
  • Grok-3 (xAI)
  • DeepSeek-V3 (DeepSeek)
• Languages: 12 languages representing five major language families (Indo-European, Afro-Asiatic, Turkic, Sino-Tibetan, Japonic):
  • WEIRD/High-Resource: English, German, Dutch, Spanish, Italian, Russian.
  • Non-WEIRD/Low-Resource: Arabic, Turkish, Chinese, Japanese, Greek, Catalan.
• Task: An adapted version of the Dentella et al. (2024) language comprehension benchmark.
  • Stimuli: 40 short scenarios (1–2 sentences) followed by a Yes/No comprehension question.
  • Design: Sentences used proper names instead of pronouns to avoid resolution ambiguity, focusing on logical inference and syntactic processing (e.g., passive voice, negation, coordination).
  • Adaptation: Native-speaking linguists translated the benchmark, ensuring grammaticality and naturalness while minimizing structural changes.
• Procedure:
  • Human Baseline: 480 native speakers (40 per language) recruited via Prolific.
  • LLM Testing: Each model was prompted 40 times per language (40 pseudo-participants) to match human sample sizes.
  • Repetition: To measure stability, every item was repeated 3 times per participant/model.
• Metrics:
  • Accuracy: Binary correctness (1 = correct, 0 = incorrect).
  • Stability: Consistency of the answer across the three repetitions (1 = consistent, 0 = inconsistent).
• Statistical Analysis: Generalized Linear Mixed-Effects Models (GLMMs) were used to analyze accuracy and stability, accounting for random effects (items, participants). Post-hoc Tukey-adjusted pairwise comparisons were conducted.
• Predictive Factors: The study modeled performance against three variables:
  1. Language Size: Number of speakers (Ethnologue data).
  2. Writing System: Latin-based vs. Non-Latin-based.
  3. Linguistic Distance: Lexical and grammatical distance from English and Spanish.

3. Key Contributions

• Challenging the "English Dominance" Hypothesis: The study provides empirical evidence that English is not the best-performing language for LLMs in comprehension tasks. In fact, Romance languages (Spanish, Italian) and Slavic languages often outperformed English.
• Decoupling Accuracy and Stability: The research highlights that accuracy (getting the right answer) and stability (consistency across repetitions) are distinct metrics. While humans generally outperformed models in accuracy, models (specifically Grok-3 and DeepSeek-V3) often exceeded humans in stability, likely due to the absence of human cognitive fatigue or attentional lapses.
• Identification of the "Romance Puzzle": The authors discovered a counter-intuitive trend where Spanish and Italian consistently achieved near-ceiling performance across all models, often surpassing English and German.
• Script and Data Interaction: The study quantifies how the interaction between writing system and data size affects performance, showing that non-Latin scripts require significantly larger datasets to achieve comparable accuracy.

4. Key Results

A. Accuracy vs. Human Baselines

• General Trend: Humans significantly outperformed all LLMs across almost all languages.
• Exceptions: Models approached human-level accuracy only in two specific cases:
  • GPT-4o in Spanish (Odds Ratio = 1.17, not significant).
  • DeepSeek-V3 in Italian (Odds Ratio = 1.07, not significant).
• Lowest Performance: Japanese and Greek consistently showed the lowest accuracy across all models, likely due to the combination of non-Latin scripts and data limitations.

B. Cross-Linguistic Variation (The "Romance Puzzle")

Contrary to the assumption that English is the "core" language of LLMs:

• English Ranking: English ranked 6th for GPT-4o and 8th for DeepSeek-V3. It was only in the top 5 for Grok-3.
• Top Performers: Spanish and Italian were the most robust languages.
  • GPT-4o: Best in Spanish, followed by Catalan and Italian.
  • DeepSeek-V3: Best in Italian, followed by Spanish, Russian, and Turkish.
  • Grok-3: Showed the least variation, with Spanish, Italian, Russian, English, and Dutch forming a top tier.

C. Drivers of Performance (RQ3)

Statistical modeling revealed three consistent drivers of variation:

1. Distance from Spanish: A significant negative correlation. Languages closer to Spanish performed better.
2. Distance from English: The effect was weak or even positive (meaning languages further from English sometimes performed better), contradicting the "English-centric" hypothesis.
3. Script & Size Interaction: The impact of language size was significantly stronger for non-Latin scripts (e.g., Greek, Japanese, Arabic) than for Latin-based scripts. This suggests non-Latin languages are more sensitive to data scarcity.

D. Stability Results

• Human vs. Model: Humans were less stable than models in most languages.
  • Grok-3 and DeepSeek-V3 were significantly more stable than humans in 9 and 7 languages, respectively.
  • GPT-4o was the only model consistently less stable than humans.
• Variation: Stability varied less across languages for models than for humans, suggesting models are less susceptible to language-specific cognitive load variations but highly sensitive to temperature settings and internal architecture.

5. Significance and Implications

• Equity in AI: The findings challenge the notion that "WEIRDness" guarantees reliability. Speakers of high-resource languages (like English) may face higher risks of misinterpretation than speakers of Romance languages, while speakers of non-WEIRD, non-Latin languages face compounded risks due to script and data scarcity.
• Tokenization Hypothesis: The authors suggest that the superior performance of Romance languages may be linked to tokenization efficiency. If models process Romance languages with fewer tokens per unit of information compared to English or German, they may retain more context and perform better on comprehension tasks.
• Future Research Directions: The study calls for moving beyond English-centric benchmarks and developing evaluation metrics that account for tokenization efficiency, script types, and the specific linguistic structures of low-resource languages.
• Safety and Reliability: Since LLMs are increasingly used in high-stakes domains (healthcare, law), the finding that they fail to reach human comprehension baselines even in high-resource languages highlights a critical safety gap that must be addressed before widespread deployment.

In conclusion, the paper demonstrates that LLMs do not possess a uniform "multilingual mind." Instead, their performance is a complex function of training data distribution, tokenization mechanics, and linguistic distance, with the surprising result that English is not the gold standard for comprehension, and Romance languages currently hold a performance advantage in these models.