💬 NLP

MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluation

This paper introduces MGSM-Pro, a multilingual extension of the GSM-Symbolic benchmark that evaluates mathematical reasoning robustness across nine languages by generating multiple digit-varying instantiations, revealing significant performance drops in low-resource languages and varying model resilience to such perturbations.

Original authors: Tianyi Xu, Kosei Uemura, Alfred Malengo Kondoro, Tadesse Destaw Belay, Catherine Nana Nyaah Essuman, Ifeoma Okoh, Ganiyat Afolabi, Ayodele Awokoya, David Ifeoluwa Adelani

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tianyi Xu, Kosei Uemura, Alfred Malengo Kondoro, Tadesse Destaw Belay, Catherine Nana Nyaah Essuman, Ifeoma Okoh, Ganiyat Afolabi, Ayodele Awokoya, David Ifeoluwa Adelani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher giving a math test to a class of students from different countries. You want to see who is truly good at math and who is just memorizing the answers from a practice book.

This paper introduces a new, tougher way to test Large Language Models (AI) on math problems, called MGSM-Pro. Here is the story of what they found, explained simply.

The Problem: The "Practice Book" Trap

For a long time, AI models have been getting better at solving math word problems. But researchers noticed something suspicious: the AI might not actually be "thinking" through the math. Instead, it might be memorizing specific details from the test questions it saw during training.

Think of it like a student who memorizes the answer to a specific word problem: "If John has 5 apples and buys 3 more..." The student knows the answer is 8. But if you change the question to "If Sarah has 5 oranges and buys 3 more...", the student who only memorized the first one might get confused.

Recently, researchers found that even in English, if you change the names or numbers in a math problem slightly, many AI models fail. They realized that the old tests were too easy because they didn't check if the AI could handle these small changes.

The Solution: MGSM-Pro (The "Shuffle" Test)

The authors created a new dataset called MGSM-Pro. They took 250 math problems and turned them into 1,240 problems (5 versions of each).

They did this by shuffling the deck:

Changing Names: Swapping "John" for "Zainabu" or "Carla."
Changing Numbers: Swapping "5 apples" for "7 apples."
Adding Distractions: Adding a sentence that sounds important but is actually nonsense (like "The sky is blue, so John bought 5 apples").

They did this in nine different languages, ranging from widely spoken ones like English and Chinese to languages with fewer digital resources like Swahili, Yoruba, and Igbo.

The Big Discovery: The "Low-Resource" Struggle

When they ran the tests, they found a massive gap between "High-Resource" languages (like English) and "Low-Resource" languages (like Twi or Igbo).

The High-Resource Students: When you changed the names or numbers in English or Chinese, the AI models dropped a little bit in score, but they stayed mostly on track. They were like a student who actually understood the math concept.
The Low-Resource Students: When they did the same shuffling for languages like Twi or Igbo, the AI models crashed. Their scores plummeted. It was as if the student suddenly forgot how to count because the teacher used a different name for the fruit.

The Analogy: Imagine a student who is great at math in English but terrible at math in their native language. When the teacher asks a question in English with a twist, the student solves it. But when the teacher asks the exact same twisted question in the student's native language, the student freezes. The paper found that AI models are doing the same thing: they are much less "robust" (reliable) in languages they haven't seen as much data for.

Who Passed and Who Failed?

The paper tested many different AI models (some made by big tech companies, some open-source).

The Robust Ones: Newer, smarter models like Gemini 3.0 Pro and GPT-OSS 120B handled the changes well. They didn't panic when the numbers or names changed.
The Fragile Ones: Older models or smaller models (like Gemma 3 4B or Llama 3) fell apart. They couldn't handle the "twist."
The Size Myth: You might think a bigger brain (more parameters) always means a smarter student. The paper found this isn't true. Sometimes a huge model failed miserably, while a slightly smaller one succeeded. It's not just about size; it's about how the model was trained.

The Lesson: Don't Just Test Once

The biggest takeaway is about how we should test AI in the future.

Currently, many leaderboards (rankings of the best AI) test a model on just one version of a problem. The authors say this is like giving a student a single practice question and calling them a genius if they get it right.

Their Recommendation: To truly know if an AI is good at math, you must test it on at least five different versions of the same problem (changing the names and numbers). If the AI gets all five right, then it's truly smart. If it only gets the first one right, it's just memorizing.

Summary

The paper argues that we need to stop treating AI like it's perfect just because it passes the easy tests. By shuffling the names and numbers in math problems across different languages, they showed that:

AI is much weaker in languages with less digital data.
Many current rankings are misleading because they don't test for this "shuffling."
To get a real picture of an AI's math skills, we need to test it on multiple variations of the same problem, not just one.

1. Problem Statement

While Large Language Models (LLMs) have made significant strides in mathematical reasoning, benchmarking in multilingual settings lags behind English in both difficulty and recency. Existing multilingual benchmarks (e.g., MGSM) often rely on static translations of English problems, making them susceptible to:

Memorization: Models may overfit to specific test instances rather than learning general reasoning.
Lack of Robustness: Models may fail when presented with minor variations (e.g., different numbers or names) of the same logical problem.
Resource Disparity: Low-resource languages (LRLs) are underrepresented and often show lower performance, but it is unclear if this is due to a lack of arithmetic capability or poor linguistic robustness.

Recent work in English (GSM-Symbolic) demonstrated that models exhibit high variance when the same question is instantiated with different names and digits. However, this phenomenon had not been systematically explored across diverse multilingual settings.

2. Methodology: MGSM-Pro Creation

The authors introduce MGSM-Pro, a robust extension of the MGSM dataset designed to evaluate multilingual mathematical reasoning under varying conditions. The creation process involves two main stages:

A. Template Construction

Source: Based on 248 questions from the original English MGSM dataset (excluding 2 questions without digits).
Symbolic Templating: The authors adopted the GSM-Symbolic framework to create templates where names and digits are replaced with placeholders (e.g., {name_male}, {num_digit}).
Irrelevant Context (IC): Following GSM-IC methodology, irrelevant sentences are inserted into the problem text to test the model's ability to filter noise.
Translation & Verification: Templates were translated into nine languages using an LLM (Gemini 2.0 Flash) and then rigorously verified by native speakers. This ensured cultural relevance (e.g., replacing "Carla" with "Zainabu" for Swahili) and correct alignment of placeholders.

B. Dataset Variations

The dataset generates five distinct instantiations per original question, organized into two series:

Symbolic Series (SYM):
- SYM_N: Changes names only.
- SYM_#: Changes digits only.
- SYM_N#: Changes both names and digits.
Irrelevant Context Series (IC):
- IC_N, IC_#, IC_N#: Mirrors the SYM variations but includes an inserted irrelevant sentence.

Languages Covered: Nine typologically diverse languages categorized by resource levels (Joshi Class):

High-Resource (HRL): English, Chinese, French, Japanese.
Low-Resource (LRL): Swahili, Amharic, Igbo, Yoruba, Twi.

3. Experimental Setup

Models Evaluated: 18 LLMs, including proprietary models (Gemini 2.5/3.0 Flash, Gemini 3.0 Pro, Claude 4 Sonnet, GPT-4.1, GPT-5) and open-source models (Gemma 3, Qwen 3/3.5, DeepSeek V3, GPT-OSS, Llama 3).
Evaluation Protocol: Zero-shot inference. Models were prompted to reason in English (CoT) but were also tested in "Native Solve" mode for select models to check language dependency.
Metrics: Accuracy on the original dataset (DO) vs. the average accuracy across 5 or 10 distinct instantiations (Avg-5, Avg-10).

4. Key Results

A. Performance Drop on Variations

Digit Variation is Critical: Changing names (SYM_N) causes minimal performance loss. However, changing digits (SYM_#), especially when combined with irrelevant context (IC_#), causes a massive drop in accuracy for all models.
The "Low-Resource" Penalty: LRLs suffer significantly more than HRLs.
- Example: On Twi (LRL), open models saw a median accuracy drop of -13.5%, whereas Chinese (HRL) saw only -4.2%.
- Example: Gemini 2.5 Flash dropped from 3rd to 7th place in rankings when evaluated on 5 instances instead of 1.

B. Model Robustness Trends

Proprietary vs. Open: Newer proprietary models (Gemini 3.0 Pro) showed higher robustness to digit changes compared to older models. Among open models, GPT-OSS 120B and DeepSeek V3 demonstrated superior robustness compared to smaller models like Gemma 3 4B.
Size is Not a Guarantee: Model parameter size does not linearly correlate with robustness.
- In the Gemma family, performance degradation worsened as model size increased (4B $\to$ 27B).
- In the GPT-OSS family, larger models were more robust.
Language vs. Arithmetic: Error analysis revealed that for LRLs, the primary failure mode is linguistic misunderstanding (e.g., failing to parse the problem structure), which cascades into logical errors. Pure arithmetic error rates remained relatively stable across languages, suggesting the bottleneck is language understanding, not calculation.

C. Leaderboard Instability

Rankings based on a single instance (standard leaderboard practice) are highly unstable. When averaged over 5 instances, the relative ordering of models changes drastically, particularly for LRLs.

5. Key Contributions

MGSM-Pro Dataset: A new benchmark expanding MGSM by 5x (from 250 to 1,240 questions per language) with systematic variations in names, digits, and context across 9 languages.
Empirical Evidence of Multilingual Fragility: Demonstrated that LLMs are significantly less robust in Low-Resource Languages when faced with digit variations, a gap not captured by standard benchmarks.
Methodological Recommendation: Proposed that math reasoning evaluation should mandate at least 5 digit-varying instantiations per problem to obtain a realistic assessment of reasoning capabilities, moving away from single-instance testing.
Insight on Training: Highlighted that robustness is likely a function of training recipes and data quality rather than just model scale.

6. Significance

This paper challenges the validity of current multilingual math benchmarks. It argues that high scores on static datasets may reflect memorization or over-optimization rather than true reasoning. By introducing MGSM-Pro, the authors provide a tool to:

Identify models that genuinely understand mathematical logic versus those that rely on pattern matching.
Highlight the specific vulnerability of Low-Resource Language models, urging the community to focus on improving linguistic robustness alongside arithmetic capability.
Standardize evaluation practices to ensure that leaderboard rankings reflect stable, robust performance rather than luck on a specific set of numbers.

The authors conclude that for a model to be considered truly capable in multilingual math, it must maintain accuracy even when the surface features (names, numbers, context) of a problem are altered.