💬 NLP

Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extraction

This survey provides a comprehensive review of document parsing research by proposing a taxonomy that distinguishes between modular pipeline systems and unified Vision-Language Model approaches, while detailing key components, evaluation standards, and future challenges in building robust document intelligence systems.

Original authors: Qintong Zhang, Bin Wang, Victor Shea-Jay Huang, Junyuan Zhang, Zhengren Wang, Hao Liang, Conghui He, Wentao Zhang

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qintong Zhang, Bin Wang, Victor Shea-Jay Huang, Junyuan Zhang, Zhengren Wang, Hao Liang, Conghui He, Wentao Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, dusty library filled with millions of old books, handwritten letters, scientific papers, and financial reports. Most of these are just images of paper—they look beautiful, but a computer can't "read" them. To a computer, a page with a chart, a paragraph of text, and a math equation is just a big, messy picture.

Document Parsing is the magic process of taking that messy picture and turning it into a clean, organized, digital file that a computer can actually use. It's like hiring a super-intelligent librarian who doesn't just read the words, but understands where they are, what they mean, and how they fit together.

This paper is a massive "user manual" and "roadmap" for the technology that does this job. Here is the breakdown in simple terms:

1. The Two Main Ways to Do the Job

The authors explain that there are two main ways to build these "digital librarians":

The Assembly Line (Modular Pipelines):
Imagine a factory where a document goes down a conveyor belt.
- Station 1: A robot finds all the boxes (tables) and circles (images).
- Station 2: Another robot reads the text inside the boxes.
- Station 3: A third robot solves the math equations.
- Station 4: A final robot puts it all into a neat file.
- The Problem: If the robot at Station 1 makes a tiny mistake (like missing a table), the robots at Station 2 and 3 get confused, and the final result is garbage. It's like a game of "telephone" where the message gets distorted at every step.
The Super-Brain (Unified VLMs):
Now, imagine a single, super-intelligent AI (a Vision-Language Model) that looks at the whole page at once. It doesn't need a conveyor belt. It sees the text, the table, and the math simultaneously and understands how they relate. It writes the final digital file in one go.
- The Benefit: It's much smarter and less likely to get confused by complex layouts.
- The Catch: These "Super-Brains" are huge, expensive to run, and sometimes they get too creative (hallucinating facts) because they are so powerful.

2. The Specific Challenges (The "Tricky Parts")

The paper dives deep into the specific hard things these systems have to learn:

Layout Analysis (The Floor Plan): Before reading, the computer needs to know the "floor plan." Is that a headline? Is that a footnote? Is that a chart? It's like distinguishing between the walls, doors, and windows of a house just by looking at a photo.
The "Math" Problem: Reading text is easy for computers. Reading math is hard because a math equation isn't just a line of text; it's a 2D structure (like a tree). The paper explains how new models are learning to "see" the structure of equations, not just the symbols.
The "Table" Problem: Tables are nightmares. They have merged cells, missing lines, and text that flows weirdly. The paper reviews how computers are getting better at turning a messy grid of numbers into a clean spreadsheet.
The "Chart" and "Chemistry" Problem: Some documents have graphs (charts) or drawings of molecules (chemical structures). The paper discusses how AI is learning to turn a picture of a graph into a list of numbers, or a drawing of a molecule into a chemical code.

3. The New "Super-Brains" (VLMs)

The paper highlights a major shift happening right now. We are moving away from the "Assembly Line" and toward the "Super-Brain."

General VLMs: These are the big, famous AI models (like the ones you might have heard of) that can do everything. They are getting good at reading documents, but they aren't perfect yet.
Specialized VLMs: These are the "Super-Brains" that have been specifically trained only on documents. They are like a master chef who only cooks Italian food; they are faster, cheaper, and more accurate at that specific task than the general chef.

4. How Do We Know They Are Good? (The Report Card)

The authors also discuss how we test these systems.

Old Way: We used to check if the computer got the words right.
New Way: We need to check if the structure is right. Did it put the table in the right place? Did it keep the math formula correct? Did it read the chart data accurately?
The Benchmark: The paper introduces new "tests" (like a driver's license exam for AI) to see which models are actually ready for the real world, where documents are often blurry, folded, or handwritten.

5. The Future: What's Next?

The paper concludes with a look at the future:

Robustness: Making these systems work even if the document is crumpled, stained, or taken with a shaky phone camera.
Efficiency: Making the "Super-Brains" smaller and faster so they can run on regular computers, not just giant servers.
Reliability: Stopping the AI from making things up (hallucinations) when it's reading complex data.

The Big Takeaway

This paper is a celebration of how far we've come. We have moved from simple "text scanners" to intelligent systems that can understand the entire story of a document. As these technologies improve, they will unlock the ability for AI to read, understand, and learn from the vast ocean of human knowledge stored in our documents, fueling everything from better search engines to smarter medical research.

1. Problem Definition

Document Parsing (DP) is defined as the process of converting unstructured or semi-structured document images (e.g., scanned PDFs, photos of papers) into structured, machine-readable representations (e.g., Markdown, JSON, LaTeX, HTML).

The Gap: Unlike traditional Optical Character Recognition (OCR), which focuses solely on transcribing text, DP must preserve structural relationships, layout organization, and semantic hierarchy across heterogeneous elements (text, tables, formulas, charts, chemical structures).
The Challenge: The field has historically been fragmented into isolated sub-tasks (e.g., layout analysis, OCR, table detection). Existing surveys lack a unified view that integrates these components, particularly overlooking the recent paradigm shift driven by Vision-Language Models (VLMs) that enable end-to-end structured generation.
Goal: To bridge the gap between fragmented task-specific literature and modern unified frameworks, providing a comprehensive taxonomy and roadmap for next-generation document intelligence systems essential for Knowledge Bases and Retrieval-Augmented Generation (RAG).

2. Methodology and Taxonomy

The authors conducted a systematic literature review (screening ~860 records down to 230 high-quality papers) and proposed a two-pronged taxonomy based on architectural paradigms:

A. Modular Pipeline-Based Systems

These systems decompose document parsing into sequential, specialized sub-tasks. Each stage is optimized independently, with intermediate representations passed downstream.

Document Layout Analysis (DLA): Detects and classifies regions (text blocks, figures, tables). Evolution: Rule-based $\to$ CNN-based (Faster R-CNN, YOLO) $\to$ Transformer-based (LayoutLM, DiT) $\to$ Multimodal (integrating text and layout).
Optical Character Recognition (OCR): Text detection and recognition. Evolution: Modular (Detection + Recognition) $\to$ End-to-End Text Spotting $\to$ Integration into VLMs.
Mathematical Expression (ME) Processing: Involves detection and recognition (conversion to LaTeX). Challenges include 2D spatial layouts and hierarchical structures. Methods range from sequence-based (Encoder-Decoder) to structure-aware (Tree-based) and VLM-based approaches.
Table Detection and Recognition: Detects table regions and reconstructs internal structure (rows, columns, merged cells). Approaches include Top-down (row/column segmentation), Bottom-up (cell detection + graph reconstruction), and Image-to-Sequence (generating HTML/LaTeX directly).
Visual Element Parsing:
- Charts: Converting visual plots to data tables or code (Chart-to-Code).
- Optical Chemical Structure Recognition (OCSR): Converting molecular diagrams to SMILES/InChI strings.

B. Unified Vision-Language Models (VLMs)

These models treat document parsing as a single end-to-end multimodal learning problem, generating structured outputs directly from raw images.

General-Purpose VLMs: Large models (e.g., GPT-4o, InternVL, Qwen-VL) adapted for documents. They offer strong reasoning but often suffer from hallucinations and unstable structural organization in dense documents.
End-to-End Specialized VLMs: Models designed specifically for document parsing (e.g., GOT-OCR2.0, DeepSeek-OCR, MinerU). They use unified decoding frameworks to handle text, formulas, tables, and charts simultaneously, often employing reinforcement learning (RL) to enforce structural correctness.
Multi-Stage Specialized VLMs: Hybrid architectures (e.g., MonkeyOCR, GLM-OCR) that reintroduce controlled decomposition (Layout $\to$ Recognition $\to$ Relation) within a unified parameter space to balance efficiency, robustness, and hallucination mitigation.

3. Key Contributions

Comprehensive Taxonomy: The first survey to systematically organize the entire document parsing workflow, covering both traditional modular pipelines and emerging VLM-driven unified models.
Detailed Component Analysis: In-depth reviews of critical sub-tasks including Layout Analysis, OCR, Mathematical Expressions, Tables, Charts, and Chemical Structures, highlighting the evolution from rule-based to deep learning and multimodal approaches.
Benchmark and Metric Consolidation: A unified summary of evaluation metrics (e.g., IoU, TEDS, CDM, Edit Distance) and benchmarks (e.g., OmniDocBench, PubTabNet, CROHME), distinguishing between sub-task evaluation and holistic document parsing.
Identification of Open Challenges:
- Robustness: Handling complex layouts, distortions (skew, warping), and low-quality scans.
- Reliability: Mitigating hallucinations in VLMs and ensuring structural faithfulness.
- Efficiency: Reducing inference costs for high-resolution documents.
- Data Scarcity: The need for high-quality, diverse, and annotated datasets for training.

4. Results and Findings

Performance Trends: The paper presents performance data (Table 2) on the OmniDocBench-v1.5, showing that Specialized End-to-End VLMs (e.g., FireRed-OCR, DeepSeek-OCR 2) and Multi-Stage VLMs (e.g., PaddleOCR-VL, GLM-OCR) generally outperform General-Purpose VLMs on structure-sensitive metrics (Formula, Table, Reading Order).
- Example: FireRed-OCR achieved 92.94% Overall score, while GPT-4o scored 75.02%.
Paradigm Shift: The field is transitioning from error-prone modular pipelines (where errors cascade) to unified models. However, pure end-to-end generation struggles with long-context consistency, leading to a resurgence of hybrid multi-stage approaches that leverage VLMs for reasoning while maintaining modular control for layout and recognition.
Evaluation Evolution: Traditional string-matching metrics (Edit Distance) are insufficient for structural tasks. New metrics like Tree-Edit-Distance-based Similarity (TEDS) for tables and Character Detection Matching (CDM) for formulas are becoming standard. Benchmarks are moving toward executable unit tests (e.g., olmOCR-Bench) to verify structural correctness rather than just text similarity.

5. Significance and Future Outlook

Foundation for AI Agents: High-fidelity document parsing is a critical upstream component for Retrieval-Augmented Generation (RAG) and AI agents. Structured data extraction directly impacts the quality of knowledge bases and the reliability of downstream reasoning.
Future Directions:
- Robustness: Developing models resilient to real-world distortions (folds, shadows, low resolution).
- Data Synthesis: Creating low-cost, high-fidelity automated data pipelines to overcome the bottleneck of manual annotation.
- Application-Oriented Evaluation: Moving beyond static benchmarks to evaluation frameworks that reflect real-world industrial scenarios (e.g., RAG performance, agent task completion).
- Reasoning Integration: Enhancing models with reasoning capabilities to handle complex logical structures and domain-specific knowledge (e.g., chemistry, law).

In conclusion, this survey establishes that while VLMs have revolutionized document parsing by enabling unified generation, the future lies in hybrid architectures that combine the structural precision of modular pipelines with the semantic reasoning of large models, supported by robust, structure-aware evaluation protocols.