ArabicNumBench: Evaluating Arabic Number Reading in Large Language Models

1. Problem Statement

While Large Language Models (LLMs) have demonstrated strong capabilities in mathematical reasoning benchmarks (e.g., GSM8K, MATH), their performance in Arabic number reading remains under-evaluated and problematic. The core challenges identified are:

• Dual Numeral Systems: Arabic contexts utilize two distinct numeral systems: Eastern Arabic-Indic (used in many Arab countries) and Western Arabic (0-9, used in others). Models often struggle to distinguish or consistently apply the correct system based on context.
• Instruction Following vs. Accuracy: Existing benchmarks primarily measure whether an answer is correct, ignoring how it is generated. In production environments, models must not only calculate correctly but also adhere to strict output formats (structured output) and follow specific reasoning chains (Chain-of-Thought).
• Lack of Granular Evaluation: Previous Arabic NLP benchmarks focus on semantic understanding or factual knowledge but fail to track extraction methods or numeral system consistency.

2. Methodology

The authors introduced ArabicNumBench, a comprehensive benchmark designed to evaluate 71 LLMs from 10 different providers.

• Dataset:
  • Scale: 210 unique test cases, resulting in 59,010 total evaluations (71 models × 4 strategies × 210 cases).
  • Categories: The tasks span six semantic categories:
    1. Pure Eastern Arabic Numerals
    2. Pure Western Arabic Numerals
    3. Contextual Addresses (multi-number extraction)
    4. Contextual Dates
    5. Contextual Quantities
    6. Contextual Prices
• Models: 71 models from 10 providers (including Qwen, Mistral, OpenAI, Meta, Google, Anthropic, Cohere, DeepCogito, Microsoft, and MoonshotAI).
• Prompting Strategies: Four distinct strategies were tested:
  1. Zero-Shot: Direct question.
  2. Zero-Shot CoT: Direct question with "Think step by step" instruction.
  3. Few-Shot: 3 demonstration examples without reasoning chains.
  4. Few-Shot CoT: 3 demonstration examples with explicit reasoning chains adapted for Arabic conventions.
• Evaluation Metrics:
  • Overall Accuracy: Correctness of the numerical answer.
  • Structured Output %: The percentage of responses successfully extracted via structured methods (e.g., specific delimiters like ####, Arabic CoT markers like "the final answer," or generic keywords) versus fallback methods (e.g., extracting the last number found).
  • Format Preservation: Consistency in using the correct numeral system (Eastern vs. Western) as required by the context.

3. Key Contributions

• ArabicNumBench Benchmark: The first comprehensive benchmark specifically targeting Arabic number reading across dual numeral systems and diverse contexts.
• Novel Metrics: Introduction of Extraction Method Tracking to distinguish between models that "guess" the right number versus those that follow instructions to produce structured, parseable output.
• Identification of Capability Divide: The paper reveals a critical dissociation between numerical reasoning (getting the right number) and instruction following (formatting the output correctly).
• Provider Comparison: A granular analysis of 10 major AI providers, highlighting that high accuracy does not guarantee production readiness.

4. Key Results

• Performance Variation: Accuracy varied wildly across models and strategies, ranging from 14.29% to 99.05%.
• Impact of Prompting:
  • Few-Shot CoT was the superior strategy, achieving an average accuracy of 80.06%, which is 2.8x higher than Zero-Shot approaches (28.76%).
  • Few-Shot CoT also increased structured output generation by 4.8x compared to Few-Shot alone (34.12% vs. 7.14%).
• The "Accuracy vs. Structure" Paradox:
  • A striking finding is that many "elite" models (e.g., Gemini 2.5 Pro with 99.05% accuracy) produce predominantly unstructured output (92.86% fallback).
  • Only 6 models (8.5%) consistently achieved ≥90% structured output even under optimal Few-Shot CoT prompting. These "Elite" models include Qwen3 Max, Llama 3.1 405B, Command A, GPT-4o-mini, Claude 3 Opus, and Qwen3 235B.
• Provider Insights:
  • Google and Qwen achieved the highest single-model accuracy (99.05%) but showed lower average performance due to weaker smaller models in their families.
  • Cohere and Microsoft demonstrated more consistent performance across their model families.

5. Significance and Implications

• Redefining Production Readiness: The paper argues that numerical accuracy alone is insufficient for deploying LLMs in Arabic NLP systems. A model can be mathematically correct but fail in production if it cannot output data in a structured format (e.g., JSON, specific delimiters) required for downstream processing.
• Prompting Ceiling: The study suggests a "prompting ceiling" where no amount of prompt engineering can transform a weak instruction-follower into a strong one. Model selection is more critical than prompt engineering for tasks requiring structured output.
• Strategic Guidance:
  • Practitioners should default to Few-Shot CoT prompting.
  • Model selection should prioritize Structured Output % alongside accuracy.
  • The six identified "Elite" models represent the only currently viable options for high-reliability Arabic number processing tasks.

In conclusion, ArabicNumBench exposes a fundamental gap in current LLM evaluation: the ability to reason numerically does not equate to the ability to follow formatting instructions. This work provides the necessary baselines and metrics to bridge that gap for real-world Arabic NLP applications.