PaddleOCR-VL-1.5: Towards a Multi-Task 0.9B VLM for Robust In-the-Wild Document Parsing

1. Problem Statement

Document parsing is critical for converting unstructured documents into structured data for Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems. However, existing solutions face two primary limitations:

• Lack of Robustness to Physical Distortions: Most state-of-the-art (SOTA) models are optimized for "digital-born" or cleanly scanned documents. They struggle significantly with real-world "in-the-wild" scenarios involving extreme physical distortions such as aggressive skewing, non-rigid page warping, screen-capture moiré patterns, and erratic lighting.
• Efficiency vs. Capability Trade-off: High-performance document parsing often requires massive parameter counts (e.g., 70B+ models), making them computationally expensive and difficult to deploy in resource-constrained environments. Conversely, smaller models often lack the robustness or multi-task capabilities required for complex layouts.

2. Methodology

PaddleOCR-VL-1.5 addresses these challenges through a two-stage framework and a specialized training strategy, maintaining an ultra-compact 0.9B parameter size.

A. Architecture

The system consists of two main components:

1. PP-DocLayoutV3 (Layout Analysis):
   • Unified Architecture: Replaces the previous decoupled pointer network with a unified, end-to-end Transformer architecture based on RT-DETR.
   • Instance Segmentation & Multi-Point Localization: Instead of standard 2-point bounding boxes, it predicts pixel-accurate masks and multi-point (quadrilateral/polygon) bounding boxes. This allows precise anchoring of text regions even on skewed or warped surfaces.
   • Integrated Reading Order: Reading order prediction is embedded directly into the Transformer decoder using a Global Pointer Mechanism. It computes pairwise precedence scores between object queries to determine the logical flow in a single forward pass, eliminating cascading errors from separate post-processing steps.
2. PaddleOCR-VL-1.5-0.9B (Element Recognition):
   • Backbone: Utilizes a Native Resolution Visual Encoder and the lightweight ERNIE-4.5-0.3B language model.
   • Multi-Task Support: Handles six core tasks: OCR, Formula Recognition, Table Recognition, Chart Recognition, Seal Recognition, and Text Spotting.
   • Text Spotting Innovation: For text spotting, the model predicts text content alongside precise spatial locations using a 4-point quadrilateral representation (Top-Left, Top-Right, Bottom-Right, Bottom-Left) rather than axis-aligned boxes. Coordinates are encoded as special tokens (<LOC_0> to <LOC_1000>) to prevent tokenization fragmentation.

B. Training Recipe

• Layout Analysis: Trained on 38k high-quality, manually annotated document samples. A Distortion-Aware Data Augmentation pipeline simulates real-world deformations (skew, warping, lighting). The model is trained end-to-end with the RT-DETR backbone and reading order transformer simultaneously.
• Element Recognition:
  • Pre-training: Scaled from 29M to 46M image-text pairs, incorporating diverse multilingual documents and specific data for seal recognition and text spotting.
  • Post-training (Instruction Tuning): Uses Uncertainty-Aware Cluster Sampling (UACS) to select "hard" examples (high uncertainty) for fine-tuning, ensuring the model focuses on difficult cases like distorted seals or dense tables.
  • Reinforcement Learning: Employs Group Relative Policy Optimization (GRPO) to unify diverse label styles and improve policy robustness.

C. Benchmark Construction: Real5-OmniDocBench

To rigorously evaluate robustness, the authors constructed Real5-OmniDocBench, a new benchmark derived from OmniDocBench v1.5. It includes five distinct real-world distortion scenarios:

1. Scanning
2. Warping
3. Screen Photography
4. Illumination (Lighting variations)
5. Skew
   All images maintain a strict one-to-one correspondence with ground-truth annotations to ensure fair evaluation.

3. Key Contributions

1. SOTA Performance with Ultra-Compact Size: Achieved 94.5% accuracy on OmniDocBench v1.5 with only 0.9B parameters, outperforming massive general VLMs (e.g., Qwen3-VL-235B, Gemini-3 Pro).
2. Robustness to Physical Distortions: Introduced PP-DocLayoutV3 and the Real5-OmniDocBench benchmark, demonstrating a 14.19% absolute improvement over its predecessor in the highly challenging "Skewing" category.
3. New Multi-Task Capabilities: Successfully integrated Seal Recognition and Text Spotting (Grounded OCR) into a single 0.9B model, enabling end-to-end detection and recognition of text in unconstrained environments.
4. Efficient Inference: Optimized for high throughput using asynchronous multi-threading and frameworks like FastDeploy, vLLM, and SGLang, achieving 1.43 pages/sec on a single NVIDIA A100.

4. Experimental Results

• OmniDocBench v1.5: PaddleOCR-VL-1.5 achieved an overall score of 94.50% (up from 92.86% in v1.0), setting a new SOTA record. It led in all sub-metrics: Text-Edit (0.035), Formula-CDM (94.21%), and Table-TEDS (92.76%).
• Real5-OmniDocBench: Achieved an overall accuracy of 92.05%, significantly outperforming larger models like Qwen3-VL-235B (88.90%) and Gemini-3 Pro (89.24%). Notably, it scored 91.66% on the "Skew" subset, a massive leap from the previous 77.47%.
• Text Spotting: Outperformed all baselines (including HunyuanOCR and Rex-Omni) across 9 dimensions, achieving an overall accuracy of 86.21%.
• Seal Recognition: Achieved a Normalized Edit Distance (NED) of 0.138, drastically outperforming the 235B-parameter Qwen3-VL (0.382).
• Inference Speed: On FastDeploy, it reached 2016.6 tokens/sec, surpassing its predecessor by 18.6%.

5. Significance

PaddleOCR-VL-1.5 represents a paradigm shift in document intelligence by proving that parameter efficiency and robustness are not mutually exclusive.

• Practical Deployment: Its small footprint (0.9B) and high speed make it viable for edge devices and large-scale enterprise deployments where massive models are impractical.
• Real-World Applicability: By specifically targeting physical distortions (skew, warping, lighting), it bridges the gap between academic benchmarks and messy real-world document processing.
• Foundation for RAG: The high-fidelity parsing of complex layouts, formulas, and seals provides a superior data foundation for downstream RAG systems, ensuring that LLMs receive accurate, structured knowledge from diverse document sources.