MDPBench: A Benchmark for Multilingual Document Parsing in Real-World Scenarios

1. Problem Statement

Document parsing, the process of converting document images into structured, machine-readable text, has seen significant progress. However, current research and benchmarks suffer from three critical limitations:

• Language Bias: Existing benchmarks predominantly focus on a handful of dominant, high-resource languages (mostly Latin scripts), neglecting low-resource and non-Latin scripts.
• Scenario Limitation: Most benchmarks rely on "digital-born" or clean scanned documents. They fail to evaluate model robustness against real-world "photographed" documents, which suffer from physical deformations (bending, wrinkling), varying lighting, camera angles, and background noise.
• Evaluation Gap: There is no systematic benchmark to assess how models perform across diverse scripts and challenging photographic conditions simultaneously. This leads to models that perform well in controlled environments but fail in deployment-ready, real-world multilingual scenarios.

2. Methodology: MDPBench Construction

The authors introduce MDPBench, the first benchmark designed specifically for multilingual digital and photographed document parsing.

A. Dataset Composition

• Scale: 3,400 document images.
• Languages: 17 languages, including Latin scripts (English, German, French, etc.) and non-Latin scripts (Arabic, Hindi, Japanese, Korean, Russian, Thai, Vietnamese, Simplified/Traditional Chinese).
• Document Types: Diverse formats including academic papers, business reports, textbooks, comics, newspapers, and handwritten notes.
• Data Split:
  • Public Set (2,720 images): For community training and initial evaluation.
  • Private Set (680 images): Held back to prevent data leakage and targeted overfitting; evaluation is conducted via a submission server.

B. Data Generation (Real-World Simulation)

To simulate real-world conditions, the authors transformed 850 digital-born documents into photographed data:

• Capture Methods: Printed on paper or displayed on screens.
• Environmental Conditions: Captured indoors and outdoors.
• Degradations: Introduced physical deformations (bending, creasing, wrinkling), camera orientation variations (left, right, oblique, inverted), image degradation (blur, shadows, glare), and background variations (tables, floors, natural scenes).

C. Annotation Pipeline

A rigorous, multi-stage pipeline was employed to ensure high-quality ground truth:

1. Expert Model Labeling: Three state-of-the-art models (PaddleOCR-VL, dots.ocr, Qwen3VL) performed layout detection and recognition. A consensus-based voting mechanism selected the most similar output among the three. If consensus was low (<0.7 similarity), the proprietary Gemini-3-Pro model was used as a fallback.
2. Manual Correction: Annotators corrected layout coordinates, reading order, and element content based on strict guidelines.
3. Human Verification: An independent reviewer verified the corrected data. Failed samples were returned for revision until they met quality standards.

D. Evaluation Metrics

• Page-Level Aggregation: Unlike previous benchmarks that average scores by element type (which biases results based on document structure), MDPBench calculates metrics per page first, then averages across pages to ensure fair comparison across languages with different structural densities.
• Metrics:
  • Text & Reading Order: Normalized Edit Distance (NED).
  • Formulas: CDM (Character-based Distance Metric).
  • Tables: Tree-Edit-Distance-based Similarity (TEDS).

3. Key Contributions

1. First Multilingual Photographed Benchmark: MDPBench is the first dataset to systematically evaluate document parsing across 17 languages and diverse real-world photographic conditions.
2. High-Quality Annotations: The dataset features rigorous human-in-the-loop verification, ensuring reliable ground truth for complex scenarios like non-Latin scripts and distorted images.
3. Comprehensive Evaluation Framework: It provides a standardized protocol for evaluating both open-source and closed-source models, including general Vision-Language Models (VLMs), specialized OCR models, and pipeline-based systems.

4. Experimental Results

The authors evaluated a wide range of models, including proprietary giants (Gemini-3-Pro), open-source VLMs (Qwen, InternVL), specialized OCRs (dots.mocr, PaddleOCR-VL), and pipeline tools.

Key Findings:

• Proprietary vs. Open-Source Gap: Closed-source models significantly outperform open-source alternatives. Gemini-3-Pro achieved the highest overall accuracy (86.4%), outperforming the best open-source model (dots.mocr, 80.5%) by 7.9% in photographed scenarios.
• Photographed Document Collapse: All models suffer significant performance degradation on photographed documents compared to digital ones, with an average drop of 17.8%.
• Non-Latin Script Deficit: Performance on non-Latin scripts is consistently lower, showing an average drop of 14.0% compared to Latin scripts.
  • Example: Models like MinerU and MonkeyOCR, trained heavily on English/Chinese, generalize well to German but drop below 10% accuracy on Arabic and Hindi.
• Specific Failure Modes:
  • Reading Order: Models struggle with Right-to-Left (RTL) scripts (e.g., Arabic), often processing text left-to-right.
  • Language Drift & Hallucination: Models frequently hallucinate content, repeat outputs, or misclassify languages (e.g., recognizing Vietnamese as Chinese).
  • Script-Specific Errors:
    • Hindi: Ignoring vowel diacritics.
    • Russian: Confusing visually similar Cyrillic and Latin characters.
    • Thai: Incorrectly inserting spaces in unspaced text, breaking lexical integrity.

5. Significance and Impact

• Highlighting Real-World Gaps: MDPBench exposes the fragility of current document parsing systems when deployed in real-world, multilingual environments. It proves that high performance on clean, digital benchmarks does not translate to robustness in the field.
• Inclusive AI Development: By identifying specific failure modes in low-resource and non-Latin languages, the benchmark provides a clear roadmap for training more inclusive and equitable AI systems.
• Standard for Future Research: It establishes a new standard for evaluating the "deployment-readiness" of document parsing models, encouraging the community to move beyond clean digital data and address the complexities of the physical world.
• Resource Availability: The benchmark and evaluation server are publicly available, fostering reproducibility and further research into robust multilingual OCR and document understanding.