OCRGenBench: A Comprehensive Benchmark for Evaluating OCR Generative Capabilities

This paper introduces OCRGenBench, a comprehensive benchmark comprising 1,060 samples and 33 tasks across text-centric generation, editing, and translation, along with a unified evaluation metric called OCRGenScore, to rigorously assess and expose the current limitations of state-of-the-art models in holistic visual text synthesis.

The Gist

Imagine you are hiring a team of digital artists to create a massive, interactive library. You don't just want them to draw pretty pictures of trees and cats; you want them to be able to write books, fix torn pages, erase graffiti from street signs, and even rewrite the text on a handwritten letter without smudging the ink.

For a long time, AI image generators (like the ones that make pictures from text prompts) were great at drawing the "trees and cats," but they were terrible at the "writing." They would draw a sign that said "STOP" but the letters would look like gibberish, or they would try to erase a word from a photo and accidentally delete the whole building behind it.

This paper introduces OCRGenBench, which is essentially a massive, rigorous final exam designed specifically to test how good these AI artists are at handling text.

Here is a breakdown of the paper using simple analogies:

1. The Problem: The "Blind Spot" in AI

Previously, there were tests for AI, but they were like giving a driver's license test that only asked you to drive in an empty parking lot.

• The Old Tests: They mostly checked if an AI could draw a single word on a poster or change one word in a photo. They ignored the hard stuff: fixing a crumpled, old document, removing handwriting from a messy note, or generating a whole page of dense text.
• The Result: AI models thought they were experts at writing because they passed the easy tests, but in the real world, they were failing the hard jobs.

2. The Solution: OCRGenBench (The Ultimate Driving Test)

The authors created a new, comprehensive test called OCRGenBench. Think of this as a driving test that includes:

• Parallel parking (generating text in a specific spot).
• Driving in a blizzard (dealing with blurry or shadowed documents).
• Rearranging traffic signs (editing text in a scene).
• Restoring a vintage car (fixing old, damaged historical documents).

What's inside the test?

• 5 Types of Text: It tests everything from formal Documents (like contracts) and messy Handwriting to Street Signs, Artistic Logos, and Posters.
• 33 Different Tasks: It doesn't just ask the AI to "draw a word." It asks it to:
  • Un-crumple a folded paper.
  • Erase a signature without touching the rest of the page.
  • Translate a handwritten note into a typed document.
  • Generate a whole page of text that looks like a real book.
• The Difficulty: The test includes pages with thousands of tiny words, text in weird shapes, and bilingual content (English and Chinese). It's designed to break the AI, not just pass it.

3. The Scorecard: OCRGenScore

How do you grade an AI that is trying to draw text? You can't just look at it; you need a ruler.
The authors created a new scoring system called OCRGenScore. Imagine a report card that grades three things at once:

1. Spelling: Did it write the right words? (Accuracy)
2. Aesthetics: Does the text look like it belongs in the picture? (Quality)
3. Obedience: Did it do exactly what you asked? (Instruction Following)

4. The Results: The "Class of 2026"

The authors tested 19 of the smartest AI models currently available (both free, open-source ones and expensive, closed-source ones).

• The Verdict: Most of the models failed. The average score was below 60 out of 100.
• The Top Performers: Only two models managed to score above 70. One was a "Unified" model (a Swiss Army knife that understands and creates) and the other was a "Specialized" model (a master craftsman focused only on generation).
• The Gap: The best models are still making mistakes that a human wouldn't make.

5. What Did They Learn? (The 8 Big Problems)

The paper found that even the "smartest" AIs have some serious blind spots:

• The "Where is it?" Problem: If you ask an AI to change a word in a paragraph, it often can't find the exact word. It might change the wrong one or erase the whole sentence.
• The "Collateral Damage" Problem: When editing text, the AI often accidentally changes the background or nearby words. It's like trying to fix a typo in a book but accidentally tearing out the page next to it.
• The "Hallucination" Problem: Sometimes the AI gets confused and invents words that weren't in the prompt, or draws a person when you asked for a sign.
• The "Tiny Text" Problem: AI struggles with small, dense text. It's like trying to paint fine details with a thick marker; the letters get blurry or turn into nonsense symbols.
• The "Language Bias" Problem: The models are much better at English than Chinese. It's like a student who studied hard for the English test but barely opened the Chinese textbook.

Why Does This Matter?

This paper is a wake-up call. It tells us that while AI is getting amazing at drawing pictures, it still isn't ready to be a reliable editor, archivist, or writer.

By creating this tough, realistic test, the authors hope to force AI developers to stop playing with the easy stuff and start fixing the hard problems. The goal is to get AI to a point where you can hand it a crumpled, handwritten, shadowy, bilingual document and say, "Fix this," and it actually does it perfectly.

In short: OCRGenBench is the "final boss" level for AI text generation, and right now, most AI players are still stuck on the tutorial.

Technical Summary

1. Problem Statement

While recent State-of-the-Art (SOTA) generative models have made significant strides in visual text synthesis, existing evaluation benchmarks are inadequate for assessing their true "OCR generative capabilities" (the ability to generate, edit, and reconstruct text within images). Current limitations include:

• Narrow Scope: Most benchmarks focus only on scene text or posters, neglecting critical categories like documents, handwriting, and layout-rich text.
• Isolated Evaluation: Existing benchmarks typically evaluate Text-to-Image (T2I) generation or text editing in isolation, failing to assess holistic capabilities across OCR-related Image-to-Image (I2I) translation tasks (e.g., dewarping, deshadowing).
• Insufficient Difficulty: Current datasets lack challenging scenarios, such as high text density, extreme aspect ratios, complex fonts, and bilingual content, leading to inflated performance metrics that do not reflect real-world reliability.

2. Methodology: OCRGenBench

The authors propose OCRGenBench, a unified benchmark designed to holistically evaluate visual text synthesis across diverse scenarios and task types.

A. Dataset Construction

• Scale: Contains 1,060 high-quality, human-annotated samples.
• Structure: Samples are organized as instruction-image-Ground Truth (GT) triplets.
• Categories: Covers 5 major text categories:
  1. Document (Modern & Historical)
  2. Handwriting
  3. Scene Text
  4. Artistic Text
  5. Layout-Rich Text (Slides, Posters)
• Tasks: Encompasses 33 distinct OCR generative tasks, including:
  • T2I Generation: Generating images from text prompts.
  • Text Editing: Modifying specific text while preserving context.
  • OCR I2I Translation: Document dewarping, deshadowing, deblurring, appearance enhancement, historical restoration, handwriting removal, scene text removal/super-resolution, and style transfer.
• Complexity: Features high text density (up to 1,300+ characters), varied aspect ratios (up to 16.7:1), diverse resolutions, and bilingual content (English and Chinese).

B. Evaluation Metric: OCRGenScore

To provide a unified assessment, the authors introduce OCRGenScore, a composite metric that aggregates performance across different task types:

• T2I Generation: Uses VIEScore (LLM-as-a-Judge for semantic consistency and perceptual quality) and Accuracy Rate (AR) based on PPOCRv5 recognition.
• Text Editing: Evaluates text accuracy (AR) and visual fidelity (LPIPS) between edited and ground truth images.
• I2I Tasks: Utilizes domain-specific metrics (e.g., Document-Distortion for dewarping, MS-SSIM for restoration) and adapts VIEScore for open-ended style transfer.
• Aggregation: All metrics are normalized to a [0, 1] range and averaged to produce a final score scaled to 0–100.

3. Key Contributions

1. First Unified Benchmark: OCRGenBench is the first to unify T2I generation, text editing, and OCR I2I translation into a single framework, covering 5 text categories and 33 tasks.
2. Comprehensive Dataset: Introduces 1,060 manually curated samples featuring high difficulty, including dense text, extreme aspect ratios, and bilingual (EN/ZH) content, addressing the "blind spots" of previous benchmarks.
3. Unified Metric: Proposes OCRGenScore, enabling fully automated, holistic evaluation of text accuracy, structural consistency, and instruction following.
4. Systematic Analysis: Provides the first large-scale evaluation of 19 SOTA models (both closed-source and open-source, specialized and unified), identifying critical bottlenecks in current generative AI.

4. Experimental Results

The authors evaluated 19 models, including closed-source giants (e.g., Nano Banana Pro, GPT Image 1.5) and open-source unified/specialized models (e.g., Flux.2-dev, LongCat-Image, Qwen-Image).

• Overall Performance: Most models score below 60/100. Only two models exceeded 70:
  • Nano Banana Pro (Closed-source, Unified): 77.19
  • Flux.2-dev (Open-source, Specialized): 70.19
• Architecture Comparison:
  • Unified Models: Show high potential but suffer from a large performance gap between closed-source (Nano Banana Pro) and open-source variants (e.g., BAGEL, Janus-4o), suggesting a need for more resources/engineering in open unified models.
  • Specialized Models: Often outperform unified models in specific tasks (e.g., Flux.2-dev and LongCat-Image), indicating that specialized architectures currently offer better performance-to-complexity ratios for text synthesis.
• Task-Specific Findings:
  • T2I Generation: Models generally handle visual quality well but struggle with text accuracy (AR), especially in dense or complex font scenarios.
  • Text Editing: Performance is poor in locating target text within dense documents, leading to unintended modifications of surrounding text.
  • Restoration Tasks: Models struggle significantly with document dewarping and historical restoration, often hallucinating content or failing to preserve aspect ratios.
• Language Bias: Models perform significantly better on English than on Chinese. Several open-source models failed completely on Chinese text generation, producing garbled output.

5. Critical Limitations Identified

Through in-depth analysis, the paper identifies 8 critical limitations in current generative models:

1. Poor Text Localization: Models fail to precisely locate target text regions in dense documents, causing editing errors.
2. Inconsistency: Modifying target text often inadvertently alters non-target text or background elements.
3. Instruction Following: Models struggle with complex instructions (e.g., specific font cases, layout constraints), leading to hallucinations.
4. Dense/Long Text: Generating long sequences or high-density text results in garbled or incorrect characters.
5. Lack of World Knowledge: Models fail to understand logical constraints (e.g., author order, ruler numbering) required for restoration tasks.
6. Specialized Domain Gaps: Performance drops significantly in vertical tasks like historical document restoration compared to creative T2I tasks.
7. Language Bias: A severe prioritization of English over other languages (e.g., Chinese), limiting global applicability.
8. Coarse Encoding/Decoding: Current VAE-based mechanisms lack the fine-grained resolution to render small-scale text strokes accurately.

6. Significance

OCRGenBench establishes a robust, standardized testbed for the next generation of generative models. By shifting the focus from "natural image generation" to "visual text synthesis," it highlights that text rendering remains a fundamental, unresolved challenge in AI. The benchmark and the proposed OCRGenScore metric provide a necessary foundation for:

• Guiding future research toward models that are not only visually appealing but also semantically accurate and reliable.
• Driving the development of multilingual, fine-grained text generation capabilities.
• Bridging the gap between general-purpose generative models and specialized OCR applications, ultimately enabling more practical AI systems for real-world document processing and creative design.

The benchmark and code are publicly available at: https://github.com/NiceRingNode/Awesome-Generative-Models-for-OCR.