Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation

1. Problem Statement

The rapid advancement of Large Language Models (LLMs) has spurred significant efforts to evaluate Arabic NLP capabilities. However, the current evaluation landscape suffers from critical reliability issues:

• Translation Artifacts: Many existing benchmarks rely on direct translations from English, introducing distributional shifts and cultural misalignment.
• Lack of Quality Validation: Native Arabic resources are often released without rigorous validation, containing annotation inconsistencies, factual errors, encoding corruption, and cultural biases.
• Fragmentation and Opacity: Evaluation is scattered across isolated tasks and domains. Furthermore, evaluation scripts and per-sample inference outputs are rarely released publicly, hindering reproducibility and community-driven improvement.
• Absence of a Unified Standard: There is no unified, quality-assured leaderboard that combines native content, code evaluation, and transparent validation.

2. Methodology: The QIMMA Framework

The authors introduce QIMMA (Arabic for "Summit"), a quality-assured Arabic LLM leaderboard. The core innovation is a multi-model quality validation pipeline applied before model evaluation.

A. Dataset Curation

• Scope: QIMMA consolidates 109 subsets from 14 source datasets, comprising over 52,000 samples.
• Domains: Covers Cultural Understanding, STEM, Legal, Medical, Trust & Safety, Code Generation, and Poetry/Literature.
• Language Purity: 99% of the content is native Arabic (Modern Standard Arabic and dialects). The only exception is code evaluation, which is inherently language-agnostic.
• Task Types: Includes Multiple Choice Questions (MCQ), Generative QA, and Code Generation.

B. The Quality Validation Pipeline

To ensure reliability, QIMMA does not simply aggregate existing benchmarks. It subjects every sample to a multi-stage filter:

1. Multi-Model Automated Assessment: Two state-of-the-art LLMs (Qwen3-235B and DeepSeek-V3-671B) evaluate each sample against a 10-point rubric.
   • Criteria: Text readability, spelling, grammar, question clarity, answer alignment, factual accuracy, cultural appropriateness, and absence of harmful content.
   • Scoring: Binary scores (0 or 1) are awarded per criterion.
2. Flagging Mechanism: Samples are flagged for human review if:
   • Either model scores below a threshold (<7/10).
   • The two models significantly disagree (leveraging model diversity to reduce bias).
3. Human Review: Native Arabic speakers with cultural expertise manually review flagged samples to resolve ambiguities, check dialectal nuances, and verify cultural alignment.
4. Code Refinement: For code benchmarks (HumanEval+ and MBPP+), the pipeline focuses on refining Arabic problem statements (prompt engineering) while keeping task logic and test suites unchanged.

C. Evaluation Implementation

• Framework: Built on LightEval for consistency and reproducibility.
• Metrics:
  • MCQ: Normalized Log-Likelihood (Accuracy).
  • Code: Pass@1 (functional correctness).
  • Generative QA: BERTScore (using bert-base-arabertv02).
• Transparency: All per-sample inference outputs, error analyses, and evaluation scripts are publicly released on Hugging Face and GitHub.

3. Key Contributions

1. Quality Validation Pipeline: A reusable methodology combining automated multi-model assessment with human review to audit and refine Arabic evaluation resources.
2. Empirical Findings on Benchmark Quality: The study reveals systematic, recurring issues in established benchmarks, proving that unvalidated resources pose concrete risks to evaluation validity.
3. The QIMMA Leaderboard: The first unified, multi-domain, multi-task Arabic LLM leaderboard grounded in native content, featuring code evaluation and full transparency.
4. Taxonomy of Errors: A structured classification of quality issues found in Arabic benchmarks (e.g., missing gold answers, encoding corruption, cultural stereotypes, and misaligned output protocols).

4. Key Results and Findings

A. Benchmark Quality Analysis

The validation pipeline identified significant flaws in existing resources:

• Discard Rates: Benchmarks showed varying discard rates (e.g., 3.08% for ArabicMMLU, 2.32% for PalmX). Notably, within ArabicMMLU, specific subsets (like Law and Professional Accounting) had discard rates exceeding 80%, indicating source-level preprocessing failures rather than subject matter difficulty.
• Common Defects:
  • Answer Quality: Factually wrong gold answers, missing options, or null fields.
  • Formatting: Encoding errors causing garbled text and duplicate samples.
  • Cultural Bias: Questions reinforcing stereotypes (e.g., attributing monolithic behaviors to entire national groups) rather than testing genuine cultural competence.
  • Protocol Misalignment: In generative tasks, gold answers often failed to follow the strict output guidelines demanded of the models.

B. Model Performance (Leaderboard Results)

The paper evaluates various SOTA open-source models (Qwen, Jais, Llama, Gemma, etc.):

• Top Performer: Jais-2-70B-Chat achieved the highest overall average score (65.81), excelling in Cultural, STEM, Legal, and Safety domains.
• Runner-up: Qwen2.5-72B-Instruct (65.75) demonstrated consistent performance across MCQ and generative tasks.
• Scale vs. Performance: While a general positive correlation exists between model size and performance (especially 1B–70B), scale alone does not guarantee success. Some large models underperformed smaller, better-tuned Arabic-specific models.
• Thinking Models: Models with "thinking" capabilities (e.g., Qwen3.5) showed significant gains in reasoning-intensive tasks (Coding) but limited benefits in generative tasks not requiring multi-step reasoning.
• Domain Specifics:
  • Coding: Qwen3.5-27B outperformed larger models, suggesting reasoning capabilities are crucial for code.
  • Medical: Llama-3.3-70B-Instruct led this domain.
  • Poetry: Gemma-3-27b-it showed strong capability in understanding poetic language.

5. Significance and Impact

• Paradigm Shift: QIMMA moves the field from "aggregating resources as-is" to "quality-first curation," establishing that even well-established benchmarks require rigorous validation.
• Cultural Sensitivity: By prioritizing native content and human review, QIMMA mitigates the cultural biases inherent in translated datasets.
• Reproducibility: The public release of per-sample outputs and evaluation code sets a new standard for transparency in Arabic NLP, allowing the community to audit results and improve models iteratively.
• Future Direction: The paper argues that future Arabic benchmark development must treat annotation as the first instance of evaluation, ensuring alignment between gold standards and model output requirements.

In conclusion, QIMMA provides a robust, transparent, and culturally grounded foundation for evaluating Arabic LLMs, addressing the critical reliability gaps that have previously hindered progress in the region's NLP ecosystem.