Technical Summary: GSM-PLUS-BN

Problem Statement

The evaluation of mathematical reasoning in Large Language Models (LLMs) has historically been dominated by high-resource languages, particularly English. This disparity creates a significant barrier to the equitable deployment of AI in linguistically diverse regions like Bangladesh, where over 230 million people speak Bengali. While benchmarks exist for Bengali mathematical reasoning (e.g., BanglaMATH, SOMADHAN), they lack systematic robustness evaluations. Crucially, there is no existing research that benchmarks a perturbed Bengali mathematical dataset. Consequently, the field lacks a mechanism to assess whether models possess true comprehension of mathematical structures or merely rely on pattern recognition, especially when faced with linguistic variations, numerical changes, or adversarial inputs in Bengali.

Methodology

To address this gap, the authors introduce GSM-PLUS-BN, a novel perturbated Bengali mathematical dataset derived from the English GSM-Plus benchmark.

Dataset Construction

• Source: The dataset is built upon 1,318 seed questions from the GSM-Plus benchmark.
• Scale: It comprises 10,544 total instances, including the seed questions and 8,000+ perturbed variants.
• Perturbation Taxonomy: The dataset employs eight distinct perturbation types across five dimensions to probe model robustness:
  1. Numerical Variation: Numerical substitution, digit expansion, and integer-decimal-fraction conversion.
  2. Arithmetic Variation: Adding operations (introducing new steps) and reversing operations (inferring inputs from outputs).
  3. Problem Understanding: Rephrasing text descriptions while preserving meaning.
  4. Distractor Insertion: Adding topic-relevant but mathematically irrelevant sentences.
  5. Critical Thinking: Removing essential information to test if the model can identify missing data rather than hallucinating an answer.
• Translation Process: A hybrid approach was used involving AI-based translation followed by rigorous human refinement. Six expert translators with backgrounds in NLP and mathematics ensured linguistic fluency, cultural localization (e.g., converting currency to BDT, names to Bengali equivalents), and logical coherence.

Experimental Setup

The study evaluated six open-source LLMs via the Groq API, spanning various parameter scales and architectures (dense and Mixture-of-Experts):

• Models: Qwen3-32B, Llama-3.1-8B-Instant, Llama-3.3-70B-Versatile, Llama-4-Scout-17B-16E-Instruct, GPT-OSS-120B, and GPT-OSS-20B.
• Prompting Strategies:
  • Standard Prompting: Direct problem presentation without reasoning scaffolding.
  • Chain-of-Thought (CoT) Prompting: Zero-shot prompting instructing the model to "think step by step."
• Evaluation: The models were tested on 9,000 samples (1,000 seed + 8,000 perturbed) using exact match accuracy as the primary metric.

Key Contributions

1. GSM-PLUS-BN Dataset: The first comprehensive, publicly available perturbed mathematical reasoning benchmark for Bengali, containing 10,544 question variations across eight perturbation types.
2. Robustness Analysis: A detailed evaluation of how different perturbation mechanisms (from numerical substitutions to distraction insertion) affect model performance in a low-resource language context.
3. Model Scaling Insights: An analysis of how model scale and architecture (dense vs. MoE) influence performance in Bengali mathematical reasoning, offering guidance for model selection in low-resource settings.
4. Prompting Efficacy: An evaluation of Standard vs. CoT prompting strategies in Bengali, revealing that while CoT generally improves performance, its effectiveness is highly model-dependent.

Results

• Seed Question Performance: Under Standard Prompting, GPT-OSS-20B achieved the highest seed accuracy (96.08%), followed closely by Llama-3.3-70B (95.90%) and Llama-4-Scout (95.68). Qwen3-32B performed poorly (13.98%) without CoT.
• Impact of CoT: CoT prompting substantially improved performance for most models, particularly Qwen3-32B, which saw a massive increase from 13.98% to 76.25% on seed questions. However, larger models like Llama-3.1-8B-Instant and GPT-OSS-120B showed performance declines or negligible gains under CoT, suggesting they may be optimized for direct answer generation.
• Perturbation Resilience:
  • Most Robust: Models demonstrated the highest resilience to Problem Understanding and Reversing Operation perturbations (often >85% accuracy with CoT).
  • Least Robust: Critical Thinking perturbations proved universally difficult, with accuracy rates remaining below 34% across all models even with CoT. Adding Operation and Integer-Decimal-Fraction Conversion also consistently yielded the lowest accuracy rates (<80%).
• Performance Gap: A significant performance gap exists between seed and perturbed questions. Under Standard Prompting, the gap ranged from 20% to nearly 40%. CoT narrowing this gap, but a persistent degradation remains, indicating models rely on surface-level patterns.
• Model Rankings: Llama-3.3-70B-Versatile and Llama-4-Scout emerged as the most consistent performers across perturbation types under CoT. GPT-OSS-120B showed the smallest performance gap between seed and perturbed questions under Standard Prompting, indicating high inherent robustness.

Significance

The paper positions GSM-PLUS-BN as a foundational resource for the Bengali NLP community. By providing a systematic benchmark for perturbed reasoning, it moves the field beyond simple accuracy metrics to assess true model robustness and comprehension. The study highlights that while CoT prompting can unlock latent reasoning capabilities in smaller models, current LLMs still struggle with complex logical reasoning (Critical Thinking) and arithmetic precision in Bengali, regardless of model size. The findings underscore the urgent need for specialized training strategies and architectural innovations to bridge the gap between pattern recognition and genuine mathematical reasoning in low-resource languages. The dataset is publicly available to facilitate future research and baseline comparisons.