Technical Summary: OvisOCR2

Problem Statement

Document parsing aims to convert visually rich document images into structured, machine-readable representations (typically Markdown), preserving not only text but also layout, reading order, tables, formulas, and visual regions. Current approaches generally fall into two categories:

1. Pipeline Methods: These decompose the task into separate stages (layout analysis, region-level recognition, and merging). While they currently lead public benchmarks, they complicate deployment due to multiple models and accumulate errors across stages (e.g., missed table boundaries cannot be fixed by downstream recognizers).
2. End-to-End Methods: These use a single model to generate Markdown in one pass, simplifying deployment and leveraging page-level context. However, existing end-to-end methods have historically lagged behind pipeline methods in performance on mainstream benchmarks.

The challenge is to develop a compact, end-to-end model that surpasses leading pipeline methods in accuracy while maintaining a small deployment footprint.

Methodology

1. Data Engine

To train a robust end-to-end parser, the authors constructed a data engine combining two complementary pipelines:

• Real-World Data Pipeline: Converts large-scale real document images into training data using specialized OCR parsers (PaddleOCR-VL-1.5 or MinerU2.5-Pro). The process involves:
  • Structured Parsing: Converting parser JSON outputs into a unified Markdown schema.
  • Rule-Based Normalization: Enforcing strict category validation, text block merging, formula normalization (LaTeX-style), table structure validation, and visual region encoding (bounding boxes as image tags).
  • Manual Spot-Checking: A conservative subset filtering stage where human inspectors verify text correspondence, formula accuracy, table alignment, and reading order. Low-quality subsets are discarded.
• Synthetic Data Pipeline: Generates controllable, noise-free training targets by deriving both the document image and Markdown ground truth from the same HTML source.
  • Hard Sample Mining: Identifies failure cases (e.g., complex tables, handwritten regions) to create reusable synthesis patterns.
  • Agent-Based Diversification: An agent expands initial HTML templates into diverse variations of content and structure while maintaining validity.
  • Ground Truth Generation: Markdown is serialized directly from the HTML source, ensuring perfect alignment.
  • Rendering: Images are rendered using Playwright to simulate realistic typography and layout.

2. Training Recipe

The training process for the 0.8B model (based on Qwen3.5-0.8B) involves a multi-stage pipeline:

• Supervised Fine-Tuning (SFT): Both a 0.8B and a 4B branch are trained on the mixed real and synthetic data. The 4B model is trained for a shorter duration to serve as a teacher.
• Reinforcement Learning (RL): The 4B branch undergoes RL using Group Relative Policy Optimization (GRPO).
  • Reward Design: A multi-component reward system evaluates text fidelity (normalized edit distance), formula matching (CDM), and table structure (TEDS). Rewards are computed based on verifiable structural correctness rather than simple token imitation.
  • Hard Case Construction: RL data focuses on "on-policy" hard samples where the model can occasionally produce superior outputs, avoiding trivial or impossible cases.
• On-Policy Distillation (OPD): The reward-aligned behavior of the 4B teacher is transferred to the 0.8B student.
  • Instead of direct RL on the small model (which causes instability and high KL divergence), the student generates responses, and the teacher provides token-level distribution supervision (reverse KL) on the student's top-k tokens. This stabilizes learning for long, structured outputs.
• Model Fusion: Several variants of the 0.8B model are trained with different configurations and fused via weighted parameter averaging to produce the final OvisOCR2.

Key Contributions

• OvisOCR2 Model: A compact 0.8B parameter end-to-end document parser that achieves state-of-the-art (SOTA) performance, surpassing leading pipeline methods on public benchmarks.
• Hybrid Data Engine: A novel system combining filtered real-world annotations with source-aligned synthetic data to cover long-tail scenarios (e.g., dense formulas, complex tables) while ensuring ground-truth accuracy.
• Advanced Training Strategy: A pipeline combining SFT, RL on a larger 4B branch with multi-component rewards, and on-policy distillation to transfer complex parsing behaviors to a compact model without the instability of direct RL on small models.

Results

OvisOCR2 was evaluated on three benchmarks:

1. OmniDocBench v1.6: OvisOCR2 achieved an overall score of 96.58, setting a new SOTA. It outperformed the best pipeline methods (e.g., PaddleOCR-VL-1.6 at 96.33) and previous end-to-end methods, leading in text edit distance, formula CDM, and table TEDS metrics.
2. PureDocBench: The model achieved the highest Avg3 score of 75.06, ranking first on both the Clean and Digital tracks. It noted that robustness on the "Real" track (degraded images) remains an area for improvement compared to massive general VLMs.
3. In-House Benchmark: Evaluated on over 1,000 pages covering long-tail scenarios (handwriting, complex tables). OvisOCR2 achieved the highest overall score (85.54) and led across all difficulty tiers. Notably, it significantly reduced the missing rate for complex tables (7.96% vs. 13–17% for pipeline methods) and achieved the best performance on handwriting subsets.

Significance

The paper claims that OvisOCR2 demonstrates the potential of end-to-end document parsing to surpass traditional pipeline methods, even with a compact 0.8B parameter model. By successfully integrating a sophisticated data engine and a multi-stage training recipe (SFT + RL + Distillation), the authors show that a single model can handle the full complexity of document parsing—text, formulas, tables, and layout—more effectively than decomposed systems. This work highlights that end-to-end approaches can now achieve deployment efficiency without sacrificing accuracy, potentially simplifying the infrastructure required for document digitization.