Technical Summary: UrduBench

Problem Statement

Despite rapid advancements in Large Language Models (LLMs) regarding reasoning capabilities, evaluating these models in low-resource languages like Urdu remains a significant challenge. Existing evaluation resources for Urdu primarily focus on Natural Language Processing (NLP) and Natural Language Understanding (NLU) tasks such as machine translation, sentiment analysis, and basic classification. These evaluations fail to assess complex reasoning abilities, including mathematical problem-solving and commonsense inference. Furthermore, direct machine translation of established English benchmarks often introduces substantial errors and contextual fragmentation, distorting evaluation outcomes and preventing fair, rigorous comparisons across multilingual models.

Methodology

To address these gaps, the authors propose UrduBench, a standardized reasoning benchmark suite for Urdu. The core of their methodology is a Contextually Ensembled Translation Framework with Human-in-the-Loop validation, designed to preserve semantic fidelity and structural integrity during the translation process.

The framework operates through the following stages:

1. Contextual Translation: Instead of translating questions and answer choices independently, the pipeline concatenates the question stem with all associated options before translation. This prevents semantic fragmentation, particularly for multiple-choice datasets like CommonSenseQA and OpenBookQA.
2. Ensemble Generation: Four complementary translation models (IndicTrans2, NLLB, Qwen-3-30B, and Gemini-2.5-Pro) generate initial translations.
3. Candidate Fusion: GPT-5.1 is employed to analyze the candidate translations against the original English source. It synthesizes an improved translation or selects the best-performing output by leveraging complementary strengths in lexical choice, grammar, and technical content handling.
4. Heuristic Post-Processing: Automated checks validate translations for issues such as empty outputs, missing Urdu characters, excessive residual English, or invalid answer keys.
5. Human Validation: Professional native Urdu-speaking annotators fluent in English perform a final review, selecting the most accurate and natural translation for each instance to resolve ambiguities and ensure cultural nuance.

Using this framework, the authors translated four widely adopted reasoning benchmarks:

• MGSM: Multilingual Grade School Math (arithmetic reasoning).
• MATH-500: Symbolic mathematics across five difficulty levels (L1–L5).
• CommonSenseQA: Commonsense knowledge and inference.
• OpenBookQA: Factual and retrieval-based scientific reasoning.

Key Contributions

• UrduBench Dataset: The creation of the first standardized, high-fidelity Urdu reasoning benchmark suite, comprising 250 MGSM, 500 MATH-500, 1,140 CommonSenseQA, and 500 OpenBookQA test instances.
• Translation Framework: A scalable, context-aware ensemble translation pipeline that integrates multiple LLMs with human-in-the-loop validation, ensuring linguistic coherence and reducing translation-induced noise.
• Comprehensive Evaluation: A large-scale evaluation of diverse state-of-the-art LLMs (ranging from 1B to 14B+ parameters), including both reasoning-oriented and instruction-tuned models, across multiple prompting strategies (Direct, Chain-of-Thought, Few-Shot + CoT).

Results and Analysis

The evaluation reveals several critical insights into multilingual reasoning:

• Task Sensitivity: Reasoning-intensive benchmarks (MGSM, MATH-500) exhibit significantly higher performance variance and lower accuracy compared to commonsense tasks (CommonSenseQA, OpenBookQA), indicating that multi-step and symbolic reasoning are particularly sensitive to linguistic transfer and translation noise.
• Difficulty Scaling: Performance on MATH-500 declines monotonically as problem difficulty increases. While larger models (e.g., Gemma-3-12B-it) and reasoning-oriented models (e.g., DeepSeek-R1-Distill-Qwen-14B) maintain relatively stronger performance at higher difficulty levels, all models show degradation, highlighting the inherent challenges of formal mathematics in low-resource settings.
• Model Architecture and Scale: While larger models generally perform better, scale alone is insufficient. For instance, Gemma-3-4B-it outperforms several 7B and 8B models, suggesting that targeted multilingual pretraining and Urdu exposure are more critical than parameter count alone.
• Reasoning vs. Instruction Tuning: Reasoning-specialized models show clear gains on arithmetic tasks (MGSM) under Chain-of-Thought prompting but do not consistently outperform general instruction-tuned models on commonsense benchmarks. This suggests that explicit reasoning supervision alone is insufficient without strong multilingual grounding.
• Language Consistency: A strong positive correlation exists between language consistency (the model's ability to respond entirely in Urdu without code-switching) and reasoning accuracy. Models with higher language consistency, such as Gemma-3-12B-it and Llama-3.1-8B-Instruct, achieve superior performance, particularly in multi-step reasoning tasks where inconsistent language use disrupts intermediate computations.

Significance

The paper establishes a scalable methodology for standardized reasoning evaluation in Urdu, providing a reproducible framework that can be adapted to other low-resource languages. By demonstrating that stable language alignment is a critical prerequisite for robust multilingual reasoning, the work offers empirical insights into the failure modes of current LLMs in non-English contexts. The authors note that their findings underscore the necessity of combining explicit reasoning prompts, sufficient model capacity, and multilingual-aware training to achieve reliable reasoning performance in low-resource languages. The code and datasets are made publicly available to facilitate further research.