Real5-OmniDocBench: A Full-Scale Physical Reconstruction Benchmark for Robust Document Parsing in the Wild

The paper introduces Real5-OmniDocBench, the first benchmark that physically reconstructs the entire OmniDocBench v1.5 dataset across five real-world scenarios to rigorously evaluate and diagnose the performance gap of Vision-Language Models in physical document parsing.

The Gist

Imagine you have a brilliant student who has aced every single test in a perfectly sterile, climate-controlled laboratory. They can read a textbook, solve a math problem, and organize a spreadsheet with 100% accuracy. You are so impressed that you hire them to work in a real-world office.

But then, things go wrong. The student is asked to read a page from a book that is curved at the spine, a receipt crumpled in a pocket, a photo of a screen taken under a flickering lamp, or a document held at a weird angle. Suddenly, the student starts making mistakes. They can't read the text because the lines are wavy, or they get confused by the shadows.

This is exactly the problem the paper Real5-OmniDocBench is trying to solve.

The Problem: The "Perfect Lab" vs. The "Messy Real World"

For a long time, computers (specifically AI models called Vision-Language Models) have been tested on "born-digital" documents. These are images that come straight from a computer file—perfectly flat, perfectly lit, and perfectly aligned. It's like testing a car only on a smooth, empty racetrack. The car looks amazing.

But in the real world, documents are messy. They get bent, folded, photographed in the dark, or scanned with a shaky hand. The paper argues that just because an AI is great at reading a perfect PDF doesn't mean it can read a crumpled receipt from a coffee shop.

The Solution: The "Physical Reconstruction"

The researchers from Baidu and HKUST decided to build a new testing ground. They took a famous, perfect test set (OmniDocBench) containing 1,355 documents and did something very physical:

1. They printed them out: They took every single digital page and printed it on high-quality paper.

2. They messed them up on purpose: They created five different "disaster scenarios" for every single page:

   • Scanning: Like putting a page in a scanner but doing it slightly crooked or with a staple in the corner.
   • Warping: Like bending the paper into a cylinder, crumpling it, or curling the edges like a dog-eared book page.
   • Screen-Photography: Taking a photo of a document displayed on a computer or phone screen (which creates weird patterns called "moiré").
   • Illumination: Taking photos in the dark, with a flashlight shining too bright, or with weird colored shadows.
   • Skew: Holding the camera at a crazy angle, making the document look like a trapezoid.

3. They kept the "Answer Key": This is the magic part. Because they started with the perfect digital version, they still knew exactly what the text should say. This allows them to compare the AI's answer on the "messy photo" against the "perfect original" to see exactly how and why it failed.

The Big Discovery: Small is Sometimes Better

The researchers tested 15 different AI models on this new, tough benchmark. They found something surprising that challenges the idea that "bigger is always better."

• The Giants: Huge, general-purpose AI models (like the ones with hundreds of billions of parameters) did okay, but they often stumbled when the paper was bent or the lighting was bad. They are like a genius who knows everything but gets confused by a messy room.
• The Specialists: A smaller, specialized model called PaddleOCR-VL-1.5 (which is tiny compared to the giants) actually won. It handled the crumpled paper, the weird angles, and the bad lighting better than the massive models.

The Analogy: Think of the giant models as a Swiss Army Knife with a thousand tools. It's impressive, but if you need to cut a specific type of rope, it might be clumsy. The small model is like a high-quality, specialized pair of scissors. It does one thing (reading messy documents) incredibly well because it was trained specifically for that job.

Why This Matters

This paper is like a "stress test" for the future of AI. It tells us that:

1. Current AI is fragile: If you build a system that only works on perfect digital files, it will fail in the real world.
2. We need better training: To make AI truly useful, we need to train it on messy, physical data, not just perfect digital files.
3. Size isn't everything: You don't need a supercomputer to solve real-world problems; sometimes, a smart, specialized tool works better.

In short, Real5-OmniDocBench is a mirror held up to the AI community, showing them that while their models are perfect in the lab, they still have a lot of work to do before they can handle the messy, unpredictable reality of our physical world.

Technical Summary

1. Problem Statement

While Vision-Language Models (VLMs) have achieved near-perfect scores on digital document benchmarks like OmniDocBench, their performance in real-world physical environments remains largely unverified. Current benchmarks suffer from a "reality gap":

• Born-Digital Bias: Existing datasets rely on pristine, distortion-free digital images, failing to account for physical perturbations encountered in deployment (e.g., curved book pages, screen moiré patterns, uneven lighting, and perspective skew).
• Lack of Controlled Evaluation: Previous "in-the-wild" datasets (e.g., WildDoc) lack digital ground-truth correspondence, making it impossible to isolate specific failure causes. Conversely, partial physical simulations (e.g., DocPTBench) use coarse sampling that cannot capture the full spectrum of real-world distortions or enable rigorous causal analysis.
• The Gap: There is no benchmark that combines the realism of physical capture with the precision of a one-to-one digital mapping to diagnose exactly which physical factor causes model degradation.

2. Methodology

The authors introduce Real5-OmniDocBench, a benchmark that performs a full-scale, one-to-one physical reconstruction of the entire OmniDocBench v1.5 test set (1,355 images).

A. Data Construction & Acquisition

• Full-Scale Reconstruction: Every digital sample is physically reproduced to create 5 variants, resulting in 6,775 total test samples.
• Ground Truth Inheritance: The benchmark strictly inherits the original JSON annotations (layout, tables, formulas, text, reading order) from OmniDocBench without modification, ensuring strict comparability.
• Acquisition Pipeline:
  • Printing: Professional Canon C5840 printer at 1200 dpi to ensure source fidelity.
  • Scale-Aware Strategy: Documents are printed in A3/A4 formats based on native dimensions to preserve font ratios and layout density.
  • Capture Devices: A heterogeneous matrix of mobile devices (Apple, Xiaomi, OPPO) is used to capture diverse ISP characteristics, sensor noise, and lens distortions.

B. Five Critical Physical Scenarios

The benchmark systematically models five orthogonal physical factors, each with multiple sub-conditions:

1. Scanning: Simulates digitization artifacts (Standard, Low-quality, Slanted, Stapled, Bound).
2. Warping: Simulates non-rigid deformations (Folding, Cylinder, Crumpling, Corner, Book spine).
3. Screen-Photography: Simulates secondary capture from digital terminals (Office Monitor, Pro Display, Laptop, Tablet, Mobile).
4. Illumination: Simulates lighting challenges (Low-light, Shadow, Color-cast, Flashlight, Refraction).
5. Skew: Simulates 3D perspective distortions from handheld capture (Pitch, Roll, Yaw, Compound, Extreme).

C. Quality Control

A multi-stage pipeline ensures data reliability:

• Machine-Assisted Detection: SOTA VLMs flag samples with catastrophic performance drops (e.g., zero scores) to identify hardware failures.
• Expert Manual Verification: Three rounds of inspection ensure semantic correspondence, structural integrity, and geometric orthogonality.
• Fidelity vs. Degradation: The protocol filters out "unusable" errors (e.g., severe blur) but retains authentic real-world artifacts (e.g., moiré patterns, minor noise) to test true robustness.

D. Evaluation Metrics

The benchmark adopts the OmniDocBench multi-dimensional scoring framework:

• Text & Reading Order: Normalized Edit Distance (NED).
• Formula Parsing: Character Detection Matching (CDM) using computational graphs for structural equivalence.
• Table Structure: Tree Edit Distance-based Similarity (TEDS).
• Overall Score: A weighted average of Text, Table, and Formula metrics.

3. Key Contributions

1. First Full-Scale Physical Benchmark: The first dataset offering a complete 1,355-image physical reconstruction with strict one-to-one digital correspondence, enabling causal attribution of performance degradation to specific physical factors.
2. Comprehensive Multi-Scenario Modeling: A systematic construction of five orthogonal physical scenarios with diverse sub-conditions, allowing for granular diagnosis of model robustness (e.g., distinguishing between scanning artifacts and handheld warping).
3. Counter-Intuitive Empirical Insights: A large-scale study of 15 state-of-the-art models revealing that compact, domain-specialized VLMs often outperform massive generalist models under physical stress, challenging the prevailing "scaling law" paradigm for document parsing.

4. Experimental Results

The authors evaluated 15 models, including General VLMs (Qwen3-VL, Gemini-3 Pro), Specialized VLMs (PaddleOCR-VL, MinerU), and Pipeline Tools.

• Specialized Models Outperform Generalists:
  • PaddleOCR-VL-1.5 (0.9B parameters) achieved the highest Overall Score of 92.05, significantly outperforming the 235B parameter Qwen3-VL-235B (88.90) and Gemini-3 Pro (89.24).
  • PaddleOCR-VL-1.5 maintained consistent high performance across all scenarios (91.25% in Warping to 93.43% in Scanning).
• Scaling Limitations: Increasing parameter count improves character recognition but does not inherently resolve geometric ambiguities (e.g., 3D perspective transformations). Large models showed fluctuations in Screen-Photography and Skew scenarios.
• Pipeline Tool Failures: Traditional heuristic-based tools (e.g., PP-StructureV3, Marker) suffered catastrophic drops in Skew scenarios (dropping to ~38-41%) due to their inability to handle large-angle perspective distortions.
• Scenario-Specific Insights:
  • Warping: Non-rigid deformations severely impact structural fidelity (TEDS) for general models, while specialized models better anchor cell boundaries.
  • Skew: This was the most challenging scenario. PaddleOCR-VL-1.5 maintained a robust Reading Order Error (ROE) of 0.061, outperforming larger models, suggesting it has internalized flexible perspective cues.

5. Significance

• Diagnostic Tool: Real5-OmniDocBench shifts the focus from simple leaderboard ranking to causal diagnosis, allowing researchers to pinpoint whether failures stem from geometric distortions, optical artifacts, or model limitations.
• Redefining Robustness: The results suggest that robustness to real-world distortions is not a simple function of parameter count but requires domain-specific inductive biases and training on diverse physical artifacts.
• Community Standard: By publicly releasing the dataset, the authors provide a "stress test" that establishes a demanding baseline, bridging the gap between digital perfection and real-world reliability. This encourages the development of truly resilient document intelligence systems capable of handling the "messy" physical world.