The Effect of Scripts and Formats on LLM Numeracy

Technical Summary: The Effect of Scripts and Formats on LLM Numeracy

Problem Statement
While Large Language Models (LLMs) have demonstrated impressive proficiency in basic arithmetic using standard Hindu–Arabic (HA) numerals and Western formatting conventions, their robustness when encountering non-standard representations remains underexplored. This work investigates whether LLM numeracy reflects genuine mathematical reasoning or is heavily conditioned on the specific surface forms (scripts and formatting) prevalent in their training corpora. The authors posit that deviations from these prevailing conventions—such as the use of alternate numeral scripts (e.g., Devanagari, Bengali) or non-standard decimal and grouping separators—may introduce significant performance gaps, even when the underlying arithmetic logic remains identical. Furthermore, the study examines how tokenization strategies, which determine how numbers are segmented into subword units, influence these outcomes.

Methodology
The authors introduce a controlled evaluation framework designed to isolate the effects of numeral representation from arithmetic complexity. The core of their methodology involves a dataset of 336 base arithmetic expressions (addition, subtraction, multiplication, division) with 1–8 digit integer and decimal operands.

1. Multiscript Dataset: Each base expression is translated into 20 distinct numeral scripts, ranging from high-resource systems (e.g., Perso-Arabic, Devanagari) to extremely low-resource scripts (e.g., Osmanya, Adlam). This allows for the evaluation of three tasks:

   • Script Identification: Identifying the script of a given number.
   • Numeral Translation: Converting numbers from various scripts to standard HA digits.
   • Arithmetic Computation: Performing calculations directly on numbers presented in non-HA scripts.

2. Format-Variation Dataset: The same 336 expressions are retained in HA digits but modified with six different formatting conventions. These variations alter decimal markers (period vs. comma) and digit grouping separators (commas, thin spaces, apostrophes) and patterns (e.g., 3-3-3 vs. 3-2-2), reflecting diverse international styles.

3. Models and Evaluation: The study evaluates nine LLMs, including four large models (Claude Sonnet 4.5, GPT-4o, Gemini 2.5 Pro, Llama-3.3 70B) and five smaller open-source models. Performance is measured via exact matching against ground truth. The authors also employ Generalized Linear Mixed-Effects Models (GLMER) to statistically quantify the impact of script, format, tokenization, and prompting strategies on accuracy, controlling for repeated measurements across the same problems.

Key Results
The study reveals that LLM performance is highly sensitive to representational variations, often suffering substantial accuracy drops despite identical mathematical tasks.

• Script Tax: Shifting from standard HA numerals to non-HA scripts results in an average performance drop of 66–87% across evaluated models. While large models generally perform well on script identification and translation (often >90% accuracy), their arithmetic capabilities degrade significantly in non-HA scripts. Low-resource scripts (e.g., Osmanya, Adlam) show near-zero accuracy for smaller models and significant drops for larger ones.
• Tokenization Impact: Statistical analysis confirms that the number of tokens per digit is a significant predictor of performance. Scripts that require more subword tokens per digit (fragmented representations) correlate strongly with lower arithmetic accuracy. For instance, Gemini 2.5 Pro, which tokenizes each digit individually, exhibits consistently lower performance on formatted numbers compared to models that group digits.
• Formatting Sensitivity: Even with HA digits, non-standard formatting conventions (e.g., using a period as a thousands separator or thin spaces for grouping) reduce accuracy. Formats resembling IP addresses (e.g., F2) or specific regional conventions (e.g., F3, F5) are particularly challenging, often leading to instruction-following failures where the model cannot reproduce the required output format.
• Prompting Strategies: The study demonstrates that targeted prompting can significantly narrow the performance gap.
  • Explicit Mapping: Providing an explicit mapping between non-HA digits and HA digits (e.g., "0:0, 1:1...") yields the largest improvement in arithmetic tasks.
  • Few-Shot Learning: Including examples of formatted inputs and outputs improves accuracy by 20–60%, effectively mitigating the "formatting penalty."
  • Native Language Prompts: Using the prompt language consistent with the numeral script also aids performance, though less effectively than explicit mapping.
  • Hinting: Merely stating the script name or describing the formatting rules without examples yields negligible improvement.

Significance and Claims
The authors claim that their findings highlight an overlooked challenge in multilingual numerical reasoning: the fragility of LLM numeracy under representational shifts. The paper argues that the observed arithmetic competence in LLMs is not purely algorithmic but is strongly conditioned on the prevalence of specific scripts and formats in pretraining data.

The significance of this work lies in:

1. Identifying the "Script Tax": Establishing that non-standard representations incur a systematic performance penalty, suggesting that current evaluations may overestimate model robustness in real-world, diverse linguistic environments.
2. Actionable Insights: Demonstrating that this loss of capability is not inevitable. The authors show that simple, parameter-free interventions—specifically few-shot prompting and explicit numeral mapping—can effectively bridge the gap, allowing models to adapt to unfamiliar scripts and formats without retraining.
3. Evaluation Practices: The work advocates for evaluation practices that reflect the diversity of numeric representations encountered in real-world language use, moving beyond the standard Western-centric conventions that dominate current benchmarks.

The paper concludes modestly, noting that while these prompting strategies are effective, they do not address all potential methods for improvement (such as chain-of-thought or tool use), and the study focuses on isolating the effects of representation rather than exhaustive optimization.