Technical Summary: URDUMMLU

Problem Statement

Current Large Language Model (LLM) evaluation relies heavily on English-centric benchmarks like MMLU and MMLU-PRO. While multilingual extensions exist, they predominantly depend on translated questions, which fail to capture knowledge grounded in native educational contexts, local history, religious studies, and region-specific curricula. This limitation is particularly acute for Urdu, a language spoken by over 230 million people with a rich literary and educational tradition. Existing Urdu resources are limited to reading comprehension, syntactic diagnostics, or translated reasoning tasks, leaving a gap in evaluating broad, native educational knowledge and reasoning capabilities in Urdu.

Methodology

The authors introduce URDUMMLU, the first broad-coverage, natively written MMLU-style benchmark for Urdu. The construction and evaluation process involved the following steps:

1. Data Collection and Construction

• Sources: Questions were collected from two primary sources: public Pakistani SSC (Secondary School Certificate) and HSSC (Higher Secondary School Certificate) examination PDFs, and native Urdu MCQ websites.
• Extraction Pipeline: A 16-stage pipeline was employed to process raw data. This included OCR for PDFs (using vision-language models), bidirectional text handling for Urdu script, normalization of punctuation and characters (e.g., mapping Arabic letters to Urdu equivalents), and structural cleanup to ensure consistent option formatting (A–D).
• Annotation Strategy:
  • Web-derived MCQs: 13,672 questions with existing answer keys were verified by human annotators.
  • Exam-derived MCQs: 12,759 questions extracted from exam PDFs lacked answer keys. These underwent dual human annotation by 17 native Urdu speakers. A strict consensus rule was applied: an item was retained only if both annotators selected the same valid answer without flags or "unsure" labels. Disagreements, flags, and unsure selections resulted in exclusion.
• Final Dataset: The benchmark comprises 26,431 multiple-choice questions across 26 subdomains grouped into 5 domains: STEM, Humanities, Social Sciences, Profession, and Other. It covers both standard academic subjects and region-specific content (e.g., Urdu Literature, Islamic Studies, Pakistan Studies).

2. Evaluation Protocol

• Models: 30 LLMs were evaluated, including proprietary API models (e.g., Gemini, GPT, Claude), open-weight models (e.g., LLaMA, Gemma, Qwen, DeepSeek), and Urdu-specific models (Qalb, Alif).
• Settings:
  • Zero-shot: All 30 models were tested under both English and Urdu instruction prompts, resulting in 60 evaluations.
  • Few-shot: Four open-source models were further evaluated in 1-shot, 3-shot, and 5-shot settings for both prompt languages.
• Metrics: Primary metric was accuracy (comparing generated answers to gold labels). Invalid-output rates (unparsable or malformed responses) were also tracked.

Key Results

Overall Performance

• Top Performer: Gemini-3.5-Flash achieved the highest accuracy, reaching 90.20% with English prompts and 90.34% with Urdu prompts.
• Open-Source Gap: The strongest open-source model, DeepSeek-V4-Flash, trailed significantly, scoring 82.41% (English) and 81.42% (Urdu), a gap of roughly 7.8 to 8.9 points compared to the best proprietary model.
• Baseline Models: Smaller open-source models (e.g., BLOOMZ, LLaMA-3.2-3B) performed near random chance (~25–35%), and even models specifically tuned for Urdu (Qalb-1.0-8B, Alif-1.0-8B) scored below 36%, indicating that Urdu-focused tuning alone does not ensure broad educational competence.

Domain Disparities

• STEM vs. Humanities: A significant performance gap exists between STEM and Urdu-centered Humanities. Models generally score 25–40 points higher on STEM subjects than on Humanities.
  • Example: Gemini-3.5-Flash scored 97.81% on STEM but 85.31% on Humanities under Urdu prompts.
  • Example: DeepSeek-V4-Flash dropped from 97.57% (STEM) to 67.32% (Humanities).
• Implication: Models can process Urdu sufficiently for scientific concepts but struggle with culturally grounded content like Urdu literature, language, and Islamic studies.

Prompt Language and Few-Shot Effects

• Prompt Language: Changing the instruction language from English to Urdu had a negligible effect on overall accuracy for most models (typically <1 point difference). This suggests the difficulty lies in the Urdu-specific content rather than the instruction language itself.
• Few-Shot Learning: Few-shot prompting (1–5 shots) yielded only modest gains (approx. +1.5 to +2.7 points) and did not alter model rankings or bridge the gap between open-source and proprietary models.

Output Validity

• Larger proprietary and open-source models maintained low invalid-output rates (<0.1%).
• Smaller models and Urdu-targeted models exhibited high invalid-output rates under Urdu prompts (e.g., BLOOMZ-3B at 74.2%, Alif-1.0-8B at 12.6%), indicating failures in following formatting instructions.

Significance and Claims

The paper claims that URDUMMLU provides the first rigorous, native benchmark for evaluating Urdu language understanding and reasoning. Its primary significance lies in demonstrating that:

1. Translation is Insufficient: Strong performance on English benchmarks does not reliably transfer to Urdu educational and cultural knowledge.
2. Knowledge Gaps: Current LLMs possess uneven knowledge, excelling in globally shared STEM concepts but lacking depth in regionally grounded Humanities.
3. Evaluation Necessity: The benchmark highlights the need for evaluation resources grounded in local educational materials rather than translated datasets to truly assess multilingual model capabilities.
4. Future Direction: The results motivate the need for continued pre-training and instruction tuning specifically on native Urdu educational and literary material to improve model performance in culturally specific domains.

The authors emphasize that the benchmark is limited to the Pakistani SSC/HSSC curriculum and does not cover undergraduate material, Indian Urdu curricula, or dialectal variations, but it serves as a foundational step toward more inclusive multilingual evaluation.