OCR or Not? Rethinking Document Information Extraction in the MLLMs Era with Real-World Large-Scale Datasets

1. Problem Statement

The document information extraction (DIE) industry traditionally relies on a two-stage pipeline: Optical Character Recognition (OCR) to convert document images into text, followed by a specialized model to extract structured data. While effective, this approach suffers from:

• System Complexity: Maintaining separate OCR and extraction models.
• Error Propagation: OCR errors inevitably degrade downstream extraction performance.
• Generalization Issues: Difficulty adapting to new domains or document layouts without retraining.

With the rise of Multimodal Large Language Models (MLLMs), which possess strong visual understanding and instruction-following capabilities, it remains unclear whether the traditional OCR step is still necessary. The paper investigates: Can powerful MLLMs perform document extraction directly from raw images (Image-Only) with performance comparable to or better than traditional OCR+MLLM pipelines?

2. Methodology

Dataset

The authors utilized two internal, high-quality industrial datasets (C1 from supply chain and C2 from finance) containing approximately 1,000 documents.

• Characteristics: High difficulty due to multilingual content (English, German, Spanish, etc.), complex nested structures, stacked cells, and heterogeneous headers.
• Ground Truth: Manually annotated structured labels and high-accuracy in-house OCR results (90%+ accuracy).

Evaluation Pipeline

The study benchmarked various state-of-the-art MLLMs (e.g., GPT-4o, Gemini 2.5, Claude 3.5, Llama 4) across three input modalities:

1. OCR-only: Text extracted by OCR.
2. Image-only: Raw document images fed directly to the MLLM.
3. Image + OCR: Both image and OCR text provided.

Performance was measured using the F1-score on key-value pair extraction.

Hierarchical Error Analysis Framework

To diagnose failure modes systematically, the authors proposed an automated framework leveraging LLMs:

1. Handler: Automatically logs and classifies prediction mismatches at character and semantic levels.
2. LLM Reasoning: Uses LLMs to map errors to predefined categories (e.g., text misinterpretation, layout issues) and, if necessary, incorporates raw images to resolve visual ambiguities.
3. Attribution: Clusters error reasons using BERT embeddings to identify root causes (e.g., OCR misalignment vs. model capability).

3. Key Contributions

1. Re-evaluation of OCR Necessity: The study demonstrates that for specific high-capacity MLLMs, OCR is not necessary. Image-only inputs can achieve performance comparable to, and in some cases superior to, OCR-enhanced pipelines.
2. Scaling Laws in DIE: Confirms that as MLLM size increases, information extraction performance improves. Notably, larger models show diminishing returns or even negative correlations with OCR input, suggesting they internalize robust text recognition.
3. Automated Error Analysis: Introduced a novel, automated hierarchical framework that uses LLMs to diagnose and categorize extraction errors, replacing manual failure analysis.
4. Optimization via Prompt Engineering: Demonstrated that general-purpose MLLMs lack task-specific knowledge. By refining schemas, exemplars, and instructions based on error analysis, performance can be significantly boosted without changing the model architecture.

4. Key Results

Performance Comparison (Table 1)

• OCR-only: Consistent performance across models (F1 ~66–74%), but limited by the ceiling of OCR accuracy.
• Image-only: Showed wider variance. However, top-tier models (e.g., Gemini 2.5 Flash, Nova Pro) achieved F1 scores comparable to OCR-only (70–75%).
• Image + OCR: Often resulted in diminished performance for top models. For example, adding OCR text to Gemini 2.0 Flash-Lite actually lowered performance by ~3% compared to Image-only.
• Conclusion: Powerful MLLMs can directly interpret visual layouts and text, making the explicit OCR step redundant and potentially detrimental due to schema ambiguity introduced by OCR formatting.

Scaling and Latency

• Scaling: Performance generally scales with model size. The Gemini 2.5 Flash and Nova Pro outperformed smaller variants significantly.
• Efficiency: Image-only pipelines reduce latency and cost by eliminating the OCR inference step. For instance, Gemini 2.5 Flash processed images in ~1.4s with low cost.

Error Analysis Findings

• Image-only yielded the lowest total error count.
• OCR-only suffered heavily from Schema Ambiguity (Error C): OCR often strips layout context, causing the model to misalign fields (e.g., confusing an item number with a quantity because they are adjacent in the text stream).
• Image-to-text errors (Error B) were higher in Image-only settings but were outweighed by the reduction in layout/alignment errors.

5. Significance and Implications

• Paradigm Shift: The paper challenges the industry standard of "OCR first, then extraction." It suggests that for enterprise-grade, high-capacity MLLMs, a single-stage, Image-Only pipeline is a more robust, simpler, and cost-effective solution.
• Practical Guidance: Developers should focus on prompt engineering (schema design, few-shot exemplars, and clear instructions) rather than relying on external OCR tools to boost MLLM performance.
• Future Directions: The work highlights the potential of "Vision-First" architectures in document AI and suggests that future research should focus on improving the visual encoders of MLLMs rather than integrating external OCR modules.

In summary, the authors conclude that OCR is becoming obsolete for powerful MLLMs in document extraction tasks, provided the models are sufficiently large and the prompts are carefully engineered to leverage the model's native visual reasoning capabilities.