Can OCR-VLMs Read Devanagari? A Stress-Test Benchmark and Post-Correction Study

Technical Summary: Can OCR-VLMs Read Devanagari?

Problem Statement
While modern OCR systems, ranging from classical engines to specialized OCR Vision-Language Models (OCR-VLMs) and frontier multimodal LLMs, demonstrate strong performance on English and Chinese document benchmarks, their efficacy on Indic scripts remains largely uncharacterized. Devanagari (used for Hindi and other languages) presents unique structural challenges absent in Latin and CJK text, including stacked conjunct consonants (sam.yuktākṣar), dependent vowel signs (matras) positioned in multiple directions, a connecting headline (shirorekha), the nukta diacritic, frequent code-mixing, and dual numeral systems. The paper addresses the gap in large-scale evaluation of new OCR-VLMs and frontier LLMs on Devanagari, questioning whether state-of-the-art gains on English/Chinese benchmarks transfer to Indic scripts.

Methodology
The authors constructed a controlled, multi-font, multi-condition benchmark and a real-world stress test:

• Data: The evaluation utilizes 100 Devanagari sentences from the FLORES test set (held out from training) rendered with five different fonts. Additionally, 300 real printed Devanagari images (word and short-phrase level) from the Sanskrit-OCR-Typed corpus were used to measure the gap between synthetic and real data.
• Degradation: Synthetic images were subjected to four conditions: clean, Gaussian blur, additive pixel noise, and low-DPI scaling.
• Models: Ten systems were evaluated across four families:
  • Classical: EasyOCR.
  • Open VLMs: Qwen2.5-VL-3B, Qwen3-VL-8B, and olmOCR-7B.
  • Specialized OCR-VLMs: DeepSeek-OCR and Unlimited-OCR.
  • Frontier Closed (API): Google Gemini 2.5 Flash, Anthropic Claude Opus 4.7, OpenAI GPT-5.5, and Mistral OCR.
• Metrics: Evaluation employs Character Error Rate (CER), Word Error Rate (WER), and chrF++ (character n-gram F-score). The authors note that code-point-based CER may understate structural errors due to Devanagari's grapheme complexity. Crucially, the study reports median CER and catastrophic rate (fraction of samples with CER > 50%) alongside the mean to account for rare but severe failure modes.
• Post-Correction: A byte-level ByT5-small model was fine-tuned on 6,000 (noisy OCR output, clean text) pairs to test if a lightweight corrector could recover errors from a cheap engine.

Key Results

1. Synthetic Text Masks Differences: On clean, rendered text, all ten systems cluster tightly within a chrF++ range of 91 to 98. Synthetic benchmarks fail to distinguish between classical engines, open VLMs, and specialized OCR-VLMs.
2. Fragility of Specialized OCR-VLMs: Under degradation, specialized models exhibit significant instability. DeepSeek-OCR, despite having the best median CER, suffers from rare but catastrophic repetition failures (outputs up to 71× the reference length), which inflate its mean error. Unlimited-OCR also degrades moderately. In contrast, generic VLMs (Qwen) and classical engines (EasyOCR) remain relatively stable under synthetic degradation.
3. Collapse on Real Scans: The performance gap widens dramatically on real printed scans. Nine of the ten systems collapse, with chrF++ scores spreading across a 76-point range.
   • Specialized Models Fail: DeepSeek-OCR and Unlimited-OCR drop to the bottom of the rankings (chrF++ 10.4 and 24.7, respectively), with high rates of hallucination and catastrophic errors.
   • English Performance is Not Predictive: Strong English OCR performance does not correlate with Indic performance. GPT-5.5 drops to chrF++ 58.5 (tying EasyOCR), and olmOCR-7B falls to 40.5.
   • Top Performers: Frontier closed models (Gemini, Claude) lead with chrF++ scores of 86.3 and 82.2. Notably, the open-source Qwen3-VL-8B (runnable on a single 24 GB GPU) achieves 75.2, outperforming GPT-5.5 and approaching Mistral.
4. Error Taxonomy:
   • Classical Engines (EasyOCR): Fail primarily on surface elements like numerals (transcribing Devanagari digits as Latin) and punctuation normalization.
   • VLMs: Fail primarily on structural elements, including conjuncts, matras, and nukta.
   • Look-alike Confusions: Recurring errors (e.g., ba↔va, gha↔dha) are consistent across systems and driven by visual/phonetic similarity.
5. Post-Correction Limits: A byte-level post-corrector trained on EasyOCR's error distribution improves that specific engine (chrF++ +1.2 to +1.5) but fails to transfer to other engines (Qwen, Unlimited, DeepSeek), where it is neutral or harmful.

Significance and Claims
The paper argues that synthetic-only benchmarks are misleading for Devanagari OCR, as they hide critical robustness differences that only real-world degradation reveals. The authors claim that:

• Specialized OCR-VLMs are currently the least safe choice for Indic scripts due to catastrophic repetition loops and poor generalization to real scans.
• Strong English OCR performance is a poor predictor of Indic OCR capability; models leading English benchmarks (GPT-5.5, olmOCR-7B) underperform significantly on Devanagari.
• Open models can compete with closed frontier models: Qwen3-VL-8B demonstrates that open, locally runnable models can approach the performance of top-tier closed APIs on Indic scripts.
• Methodological Contribution: The study advocates for reporting median CER and catastrophic rates rather than just the mean to faithfully summarize OCR performance, and provides a distribution-matched post-correction approach that requires engine-specific training.

The authors conclude that real evaluation is indispensable for Indic scripts and release the benchmark, code, and models to support future evaluation of OCR systems on Indic scripts.