Mind the Gap... or Not? How Translation Errors and Evaluation Details Skew Multilingual Results

Problem
Recent large language models (LLMs) have demonstrated impressive capabilities in multilingual settings, supporting both high-resource and low-resource languages. However, evaluations on standard benchmarks, specifically the Multilingual Grade School Math (MGSM) dataset, have consistently shown a significant performance gap between English and other languages. Initial observations suggested that models perform well in English but struggle significantly in languages like French, Swahili, and Telugu, even for high-resource languages. This phenomenon led to the hypothesis that LLMs lack true cross-lingual capability generalization. The paper argues that these conclusions may be skewed not by inherent model limitations, but by data quality issues within the benchmark and non-standardized evaluation practices.

Methodology
The authors conducted a comprehensive re-evaluation of the MGSM benchmark using a diverse set of LLMs, including open-source models (Gemma 3, DeepSeek V3) and proprietary systems (Gemini, Claude, GPT). The study followed a three-phase approach:

1. Baseline Replication: The authors first replicated previous experiments using the original MGSM dataset and the standard answer extraction method (a regular expression based on the Kaggle benchmark starter code). This established the "initial analysis" showing a large cross-lingual performance gap.
2. Data and Evaluation Audit: The authors performed a detailed manual and semi-automatic audit of the dataset and the evaluation pipeline. They identified two primary sources of error:
   • Translation and Formulation Errors: Subtle translation errors in the target languages (e.g., changing "Tuesday" to "Thursday" or altering price multipliers) and ambiguities in the original English questions (e.g., missing unit specifications).
   • Answer Extraction Flaws: The standard extraction method failed to account for regional formatting conventions, such as the use of dots for thousands separators and commas for decimals in French and German, or the use of native numeral systems (e.g., Bengali numerals) instead of Arabic digits.
3. Correction and Re-evaluation: The authors created a corrected version of the MGSM dataset (MGSM-Rev2) by fixing translation errors, clarifying ambiguous questions, and standardizing the source English text. They also implemented an improved, language-aware answer extraction method that handles diverse numerical formats and maps native numerals to Arabic digits. They then re-ran the experiments on this corrected corpus.

Key Contributions

• Identification of Systemic Errors: The paper reveals that the MGSM benchmark contains numerous translation inaccuracies and formatting mismatches that disproportionately affect non-English evaluation results.
• Semi-Automated Quality Assurance: The authors propose a scalable, semi-automatic method for identifying problematic questions by leveraging strong LLMs to detect inconsistencies in translation and solvability.
• Standardized Extraction Protocol: The paper highlights the critical need for language-specific answer extraction rules to handle regional number formatting and numeral systems, providing a more robust extraction function.
• Corrected Dataset Release: The authors release the corrected MGSM dataset (MGSM-Rev2) to the research community to facilitate more accurate future evaluations.

Results
The re-evaluation yielded results that starkly contrasted with the initial analysis:

• Disappearance of the Gap: After correcting the data and improving answer extraction, the performance gap between English and other languages largely disappeared for state-of-the-art models. For instance, the performance difference for French dropped from a 17.6% loss to just 1.2% for GPT5.
• Specific Improvements:
  • French: A simple adjustment to the regular expression to accommodate French number formatting yielded a 10% improvement for GPT5.
  • Bengali: Converting native Bengali numerals to Arabic equivalents in the model output led to a dramatic accuracy increase for Gemma 27B, rising from 45.2% to 91.2%.
  • General Trend: For the strongest models (e.g., Gemini 2.5 Pro, GPT5, Claude Sonnet), the cross-lingual disparity became negligible (maximum differences of 1.2% to 2.0%).
• Remaining Gaps: Small performance gaps persisted for smaller or weaker models (e.g., Gemma 12B, GPT Nano), suggesting that while data quality was the primary bottleneck for strong models, weaker models may still face limitations in cross-lingual reasoning or pre-training data exposure.

Significance
The paper serves as a critical call to action for the research community regarding the rigor of multilingual evaluation. It demonstrates that observed performance gaps in LLMs can be artifacts of poor test data quality and evaluation methodology rather than genuine model deficiencies. The authors emphasize that conclusions drawn from "just reporting numbers" without deep analysis of the test setup can be misleading. By correcting these issues, the paper suggests that current LLMs possess a much higher degree of cross-lingual capability generalization than previously believed. The work underscores the necessity of ensuring high-quality test data and standardized evaluation protocols, especially as LLMs increasingly generate the data used to train and evaluate future models. The authors conclude that while the MGSM benchmark may be saturated for top-tier models, it remains a valuable tool for testing smaller models, provided the evaluation setup is meticulously controlled.