KazakhOCR: A Synthetic Benchmark for Evaluating Multimodal Models in Low-Resource Kazakh Script OCR

1. Problem Statement

The paper addresses the critical lack of Optical Character Recognition (OCR) resources and benchmarks for Kazakh, a Turkic language spoken by 16 million people. Kazakh is unique because it utilizes three distinct writing systems depending on geography and politics:

• Cyrillic: Used in Central Asia (e.g., Kazakhstan).
• Latin: Used in Europe, America, and Turkey.
• Arabic: Used in China, Afghanistan, Pakistan, and Iran.

While OCR research exists for high-resource languages and the Kazakh Cyrillic script, there is a severe scarcity of data and benchmarks for the Kazakh Arabic and Kazakh Latin scripts. Furthermore, existing Multimodal Large Language Models (MLLMs) have not been rigorously tested on these low-resource, Abjad-based scripts. The authors hypothesize that current MLLMs may struggle with these scripts due to a lack of training data and potential hallucination issues when dealing with low-resource languages.

2. Methodology

A. Dataset Construction (KazakhOCR)

To overcome the lack of real-world data, the authors constructed a synthetic OCR benchmark consisting of 7,219 images covering all three scripts.

• Source Text:
  • Arabic Script: Sourced from the MC² corpus (the first Kazakh corpus in Arabic script).
  • Cyrillic Script: Sourced from Kazakh Wikipedia.
  • Latin Script: Generated by transliterating the Cyrillic text using the QazNLTK package.
• Balancing: Due to the Arabic script text being significantly longer than the others, the authors performed random balanced subsampling to ensure similar text length distributions across all three scripts (approx. 2,400 texts per script).
• Image Generation: Using the Python Pillow library, the authors applied realistic variations to simulate real-world OCR challenges:
  • Fonts: Random selection from 135 (Arabic), 431 (Cyrillic), and 1,376 (Latin) compatible Google Fonts.
  • Noise & Blur: Random noise levels (4–18) and blur effects (0–0.8).
  • Rotation: Random rotation between -3 and 3 degrees.
  • Color: Two palettes (light background/dark text) ensuring a minimum contrast ratio of 4.5.
• Availability: The full dataset is publicly released on Hugging Face.

B. Model Evaluation

The study evaluated three state-of-the-art Multimodal Large Language Models (MLLMs) on a subset of 600 images (200 per script):

1. Gemma-3-12B-it
2. Qwen2.5-VL-7B-Instruct
3. Llama-3.2-11B-Vision-Instruct

• Baseline: A classical OCR engine, Tesseract, was used as a baseline with its pre-trained models for Arabic, Latin, and Cyrillic scripts (no fine-tuning).
• Metrics: Character Error Rate (CER), Word Error Rate (WER), and Language Identification Accuracy (percentage of samples correctly identified as Kazakh).

3. Key Results

A. Performance by Script

• Cyrillic Script: All models performed best here, as this is the most represented script in training data.
  • Qwen2.5-VL achieved the best results with a CER of 5.3% and WER of 13.4%.
  • Tesseract performed slightly better with a CER of 4.3%.
• Latin Script: Performance dropped significantly.
  • Qwen achieved a CER of 26.4%.
  • Llama and Gemma struggled with CERs of 21.6% and 31.0% respectively.
  • Language ID: Qwen correctly identified Latin Kazakh as Kazakh 99.4% of the time, while Gemma and Llama misclassified it frequently (often as Kyrgyz or Tatar).
• Arabic Script: This was the most challenging domain.
  • CERs ranged from 35.5% (Qwen) to 72.5% (Gemma).
  • Language ID Failure: All MLLMs failed to identify the text as Kazakh. Correct identification rates were between 0.0% and 1.1%.
  • Misclassification: The models consistently misidentified Kazakh Arabic script as Arabic, Farsi, or Kurdish.
  • Baseline Comparison: Tesseract significantly outperformed MLLMs in the Arabic script (CER 15.0% vs. MLLM CERs of 35.5%–72.5%).

B. MLLMs vs. Classical OCR

• Traditional OCR (Tesseract) consistently achieved lower error rates than MLLMs across all scripts, with the gap being most pronounced in the Arabic script.
• MLLM Limitations: Despite their general multimodal capabilities, MLLMs failed to match traditional OCR accuracy on low-resource scripts. The authors suggest this is due to hallucination, where models generate characters or words from high-resource languages (like standard Arabic or Farsi) that resemble the low-resource input, leading to high error rates.

4. Key Contributions

1. KazakhOCR Benchmark: The creation of the first synthetic OCR benchmark specifically for low-resource Kazakh scripts (Arabic, Latin, and Cyrillic), filling a major gap in NLP resources.
2. Comprehensive Evaluation: The first systematic evaluation of modern MLLMs (Gemma, Qwen, Llama) on Kazakh OCR, revealing severe performance disparities based on the script used.
3. Identification of Failure Modes: The study highlights that MLLMs not only struggle with text extraction (high CER) but also fail fundamentally at language identification for the Kazakh Arabic script, misclassifying it as other languages entirely.
4. Open Source Release: Publicly releasing the dataset and code to foster reproducibility and future research in low-resource NLP.

5. Significance and Future Implications

• Inequity in AI: The results demonstrate a significant "digital divide" where MLLMs are highly capable for high-resource scripts (Cyrillic) but fail for low-resource variants (Arabic/Latin Kazakh), potentially excluding millions of speakers in Central and East Asia.
• Need for Inclusive Training: The paper argues that future LLM/MLLM training must explicitly include low-resource Abjad-based scripts to prevent hallucination and improve language identification.
• Cultural Context: The inability to recognize Kazakh Arabic script is particularly concerning for speakers in China, Afghanistan, Pakistan, and Iran, where this script is the primary medium of communication.
• Future Work: The authors suggest future research should focus on prompt engineering, using authentic (non-transliterated) Latin Kazakh texts to capture cultural nuances, and moving from synthetic to authentic real-world OCR data as it becomes available.

In conclusion, KazakhOCR serves as a critical diagnostic tool, proving that while MLLMs are powerful, they currently lack the robustness required for low-resource, multi-script languages, necessitating targeted development for equitable AI support.