Technical Summary: MonkeyOCRv2

Problem Statement

Mainstream visual foundation models (e.g., CLIP, DINO, SAM) are primarily pretrained on natural images, focusing on high-level object semantics, global alignment, and scene context. These models exhibit a representation mismatch when applied to document images. Document images are characterized by dense text, fine-grained character strokes, complex layouts, and semantics that rely heavily on local visual details (e.g., punctuation, superscripts, stroke variations). Existing models often discard these fine-grained details in favor of semantic abstraction, rendering them ill-suited for document parsing, understanding, and related tasks. Furthermore, existing document-oriented pretraining datasets are limited in scale, language diversity, and annotation granularity, often covering only Chinese and English or lacking dense supervision.

Methodology

1. Data Engine: MonkeyDoc v2

To address the data scarcity and distribution gap, the authors constructed MonkeyDoc v2, the largest known document-oriented visual-text pretraining corpus.

• Scale: 113 million images.
• Diversity: Covers 17 languages (including Simplified/Traditional Chinese, English, Arabic, German, etc.) and diverse document types (scientific papers, invoices, handwritten notes, ancient texts, formulas, tables).
• Composition: 61M real-world samples and 52M synthetic samples.
• Annotation Pipeline: Utilizes a multi-expert agreement pipeline where multiple complementary OCR models transcribe cropped elements; the prediction with the highest pairwise agreement is retained to suppress model-specific errors. Low-quality samples are filtered using layout verification and logical consistency checks via large language models.

2. Pretraining Strategy: Joint Generation and Reconstruction

MonkeyOCRv2 employs a dual-objective pretraining strategy to learn document-native visual representations:

• Image-to-Text Generation: Aligns visual representations with textual content using standard autoregressive cross-entropy loss ($L_{text}$). This provides dense supervision for semantic understanding.
• Pixel-Level Document Reconstruction: Encourages the encoder to preserve fine-grained visual details (character strokes, glyph shapes, layout structures) that might be discarded under purely textual supervision. This is achieved via a Mean Squared Error (MSE) loss ($L_{pix}$) and, optionally, a structure-aware loss ($L_{struct}$) matching edge and distance-to-edge maps.
• Combined Objective: $L_{pretrain} = L_{text} + \lambda L_{rec}$. The reconstruction objective acts as a regularizer, ensuring the encoder retains visual evidence even when linguistic context is weak or unavailable.

3. Architecture

The model family includes three vision encoder variants trained from scratch on MonkeyDoc v2:

• MonkeyOCRv2-S: ViT-Small (28M parameters).
• MonkeyOCRv2-B: ViT-Base (113M parameters).
• MonkeyOCRv2-AS: ViTAEv2-Small (21M parameters), optimized for tasks benefiting from multi-scale inductive bias (e.g., detection, segmentation).

Key Contributions

1. MonkeyOCRv2: A visual-text foundation model specifically pretrained for document AI, addressing the representation mismatch between natural-image encoders and document images through joint text generation and pixel-level reconstruction.
2. MonkeyDoc v2: A massive, multilingual (17 languages), multi-type (113M images) pretraining corpus that significantly exceeds the scale and diversity of existing document datasets.
3. Dual-Objective Pretraining: Demonstrates that coupling image-to-text generation with pixel-level reconstruction improves robustness, particularly when linguistic context is removed or degraded.

Experimental Results

Downstream Task Performance

Replacing original encoders with MonkeyOCRv2 yielded consistent improvements across five representative document analysis tasks:

• Text Recognition: Improved CRNN accuracy from 58.7% to 67.3% on challenging benchmarks. PARSeq with MonkeyOCRv2 achieved state-of-the-art (SOTA) performance on Union14M and Chinese benchmarks.
• Formula Recognition: A 110M parameter model (UniMERNet-T + MonkeyOCRv2) outperformed the 325M UniMERNet-B, demonstrating that a stronger document-oriented encoder can compensate for reduced model capacity.
• Text Detection: Consistent F-measure gains across ICDAR2015, ArT, Total-Text, and CTW1500, outperforming both ImageNet-pretrained encoders and oCLIP (a text-specific encoder).
• Document Tampering Detection: Achieved SOTA F1-scores on DocTamper-Test, DocTamper-FCD, and DocTamper-SCD, showing strong generalization across domain shifts.
• Overlapping Text Segmentation: Improved mIoU by 4.3% to 6.3% on the MOT dataset.

Document Parsing and Understanding

• Document Parsing (MDPBench): MonkeyOCRv2-B-Parsing (0.7B total parameters, 0.1B vision encoder) achieved 83.3% on MDPBench, surpassing the previous open-source SOTA (dots.mocr, 3B parameters) by 2.8% absolute, despite using a vision encoder roughly 11× smaller. It also outperformed larger general-purpose VLMs like Qwen3-VL-235B and GPT-5.2 on OmniDocBench.
• Document Understanding: In a controlled setting (frozen encoders paired with the same Qwen3-1.7B LLM), MonkeyOCRv2-B achieved an average score of 57.2 across eight benchmarks, significantly outperforming CLIP, DINO, SAM, and other document-oriented encoders.

Robustness Analysis

• Linguistic Context Dependence: A controlled study using scrambled text showed that MonkeyOCRv2 is more robust to resolution degradation and linguistic context removal. The pixel reconstruction objective narrowed the accuracy gap between semantically coherent and scrambled text, indicating a stronger reliance on visual evidence rather than language priors.
• Hallucination: On CHAOS-Bench, MonkeyOCRv2-Parsing achieved the highest page-average recall for perturbed words, demonstrating superior faithfulness to visual evidence compared to models like HunyuanOCR-1.5 and PaddleOCR-VL-1.6.

Significance and Claims

The paper posits that document-oriented visual pretraining can serve as a foundation for document intelligence in its own right, rather than relying on encoders built for natural scenes.

• Foundational Shift: The work challenges the paradigm of adapting general-purpose vision models to documents, arguing that the unique visual statistics of documents (dense text, fine-grained strokes) require dedicated pretraining objectives.
• Efficiency: It demonstrates that a compact, document-native encoder (0.1B) can outperform massive general-purpose VLMs (hundreds of billions of parameters) on document-specific tasks when the visual representation is optimized for the domain.
• Visual Evidence Preservation: The study validates that pixel-level reconstruction objectives are critical for preserving fine-grained details, improving robustness in scenarios where linguistic context is ambiguous or absent.

The authors conclude that MonkeyOCRv2 and MonkeyDoc v2 provide reusable foundations for multilingual OCR, document understanding, and broader document-intelligence systems, establishing text as a "first-class visual modality."