mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?

Technical Summary of mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?

Problem Statement
While Large Language Models (LLMs) have demonstrated strong reasoning capabilities, their performance across diverse languages remains under-explored. The current LLM ecosystem is predominantly English-centric, facing challenges such as limited data for many languages, uneven performance across language communities, and tokenizer-induced disparities that affect both effectiveness and inference cost. Existing benchmarks often rely on machine-translated items or focus on specific modalities, lacking a rigorous, controlled evaluation of reasoning stability across a broad spectrum of languages using high-quality, human-validated translations.

Methodology
The authors introduce mmPISA-bench, a compact, high-quality multilingual reasoning benchmark derived from the OECD Programme for International Student Assessment (PISA).

• Dataset Construction: The benchmark consists of 25 multiple-choice questions requiring reasoning: 11 from PISA 2022 (Mathematics) and 14 from PISA 2018 (Reading Comprehension). These items were selected based on strict criteria: availability of official human translations, exclusive reliance on text (no images/interactive elements), and multiple-choice format.
• Language Coverage: The dataset covers 43 languages. For each language, the dataset includes both the official human translation and a machine-translated counterpart (using Google Translate), resulting in a total of 2,150 data points.
• Experimental Design: Two proprietary frontier LLMs were evaluated: OpenAI's GPT-5.1-2025-11-13 and Anthropic's Claude (Opus-4-5-20251101 and Haiku-4-5-20251001). The models were tested across five reasoning effort levels (none, double-prompt, low, medium, high) and two translation types (human vs. machine). Each configuration was repeated five times to assess stability. The evaluation was conducted in a zero-shot setting with uniform system prompts enforcing multiple-choice output formats.
• Metrics: Performance was measured by accuracy, token usage (input and output), and inference cost.

Key Contributions

1. mmPISA-bench Dataset: The release of a 2,150-item dataset featuring 25 reasoning questions in 43 languages, comprising both official human translations and matched machine translations. This allows for direct comparison against 15-year-old student performance benchmarks.
2. Cross-Linguistic Reasoning Analysis: A systematic evaluation of LLM reasoning stability across 43 languages, moving beyond the English-dominated evaluation landscape.
3. Machine Translation Validity: An empirical investigation into whether machine-translated questions degrade reasoning performance compared to official human translations.
4. Cost-Accuracy Trade-off Analysis: A detailed analysis of how tokenization and reasoning length vary across languages, linking these factors to inference costs and accuracy.

Results

• RQ1 (Consistency): Modern LLMs can reason effectively across all 43 evaluated languages, achieving accuracy comparable to human test-takers. While performance variations exist, they are relatively bounded. Under high reasoning effort, Claude achieved perfect accuracy (100%) in seven languages, while GPT did so in only one (German). GPT exhibited less variability across languages than Claude, though Claude achieved higher average accuracy overall.
• RQ2 (Human vs. Machine Translation): Machine-translated questions did not degrade accuracy relative to official human translations; in several configurations, machine-translated inputs yielded marginally higher accuracy. This suggests high-quality machine translation (synthetic data) is adequate for large-scale multilingual reasoning evaluations where official translations are unavailable.
• RQ3 (Reasoning Length and Cost):
  • Tokenization Disparities: Significant variations in token counts were observed across languages due to tokenization differences.
  • Cost-Accuracy Correlation: A negative correlation was found between cost and accuracy for both models. Languages that were more expensive to process (due to higher input token counts and/or longer reasoning traces) tended to yield lower accuracy.
  • Reasoning Behavior: For Claude, there was a strong positive correlation between input and output token counts (0.950), indicating that longer inputs in certain languages elicited proportionally longer reasoning traces. GPT showed a weaker correlation (0.334), suggesting its reasoning length is less dependent on input length.
  • Cross-Language Reasoning: Qualitative analysis revealed that models sometimes perform reasoning in languages other than the input or English (e.g., Claude reasoning in Russian when prompted in Kazakh), a phenomenon detectable only with native-language expertise.

Significance and Claims
The paper claims that leading proprietary LLMs have reached a level of multilingual reasoning robustness where they can perform comparably to human students across a wide array of languages. However, the authors emphasize that "reasoning equally well" is nuanced: while accuracy is high, the efficiency and cost of reasoning vary systematically by language.

The study highlights that some languages are simultaneously more expensive and less accurate, driven by tokenization inflation and language-specific reasoning strategies. Consequently, the authors argue that multilingual evaluation must move beyond aggregate accuracy metrics to include reasoning length and cost analysis. Furthermore, the finding that machine translation does not degrade performance supports the scalability of multilingual benchmarks using synthetic data, provided the source material is high-quality. Finally, the observation of cross-language reasoning behaviors underscores the need for native-language expertise in evaluating LLMs, as aggregate metrics may mask complex internal linguistic shifts.