💻 computer science

Reading or Reasoning? Format Decoupled Reinforcement Learning for Document OCR

This paper proposes Format Decoupled Reinforcement Learning (FD-RL), a novel approach that leverages entropy-based data filtration and format-specific rewards to address high output uncertainty in complex document OCR, achieving a new state-of-the-art score of 90.41 on the OmniDocBench benchmark.

Original authors: Yufeng Zhong, Lei Chen, Zhixiong Zeng, Xuanle Zhao, Deyang Jiang, Liming Zheng, Jing Huang, Haibo Qiu, Peng Shi, Siqi Yang, Lin Ma

Published 2026-01-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yufeng Zhong, Lei Chen, Zhixiong Zeng, Xuanle Zhao, Deyang Jiang, Liming Zheng, Jing Huang, Haibo Qiu, Peng Shi, Siqi Yang, Lin Ma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to read a messy, complicated document. This document isn't just a simple letter; it's a mix of plain paragraphs, complex math equations, and tricky tables.

For a long time, researchers thought reading was just about "seeing" the words clearly. They tried to feed the robot more and more examples (data engineering) to help it memorize what things look like. But the authors of this paper noticed a problem: The robot gets very confused when the document has a lot of structure.

Here is the simple breakdown of their discovery and solution:

1. The Problem: The Robot's "Confusion Meter"

The authors found that when a robot reads simple text, it is very confident. But when it sees a math formula or a table, its "confidence meter" (which they call entropy) drops drastically. It's like a student who can easily recite a poem but freezes up when asked to solve a calculus problem or organize a spreadsheet.

The robot tries to guess the next word, but because there are so many ways to write a formula or arrange a table, it gets lost. It starts guessing randomly, which leads to errors.

2. The Solution: "Format Decoupled Reinforcement Learning" (FD-RL)

Instead of just forcing the robot to memorize more examples, the authors taught it to think about the structure. They call their method FD-RL.

Think of it like training a chef:

Old Way (SFT): You give the chef a million recipes and say, "Memorize these." The chef learns to copy the recipes perfectly but might mess up if you ask for a dish with a slightly different ingredient list.
New Way (FD-RL): You teach the chef to understand why a cake rises or why a sauce thickens. You give them specific rules for different types of cooking.

3. How They Did It (The Two-Step Training)

Step 1: The "SFT" (Supervised Fine-Tuning) - Learning the Basics
First, they fed the robot a massive library of documents (books, PDFs, synthetic data) to teach it how to read text, formulas, and tables in general. This is like the robot learning its ABCs and basic grammar.

Step 2: The "RL" (Reinforcement Learning) - The Specialized Coaching
This is where the magic happens. They used two clever tricks to fix the robot's confusion:

Trick A: The "Hard Mode" Filter (Entropy-Based Filtration)
Instead of training the robot on every document, they used a filter to pick out only the hardest, most confusing documents (the ones with high "entropy").
- Analogy: Imagine a math teacher who stops giving the student easy addition problems and only gives them complex calculus problems. By focusing only on the hard stuff, the student learns to reason through the confusion rather than just guessing.
Trick B: The "Specialized Judge" (Format Decoupled Rewards)
In the past, the robot was graded with one single score: "Did you get the words right?" If the robot wrote the right words but put them in the wrong table structure, it still got a bad grade, but the robot didn't know why.

The authors changed the grading system. They gave the robot three different judges:
1. The Text Judge: Checks if the plain words are spelled right.
2. The Math Judge: Checks if the formulas make mathematical sense (even if the symbols look slightly different).
3. The Table Judge: Checks if the table rows and columns are aligned correctly.
Analogy: If you are building a house, you don't just ask, "Is the house built?" You have a plumber check the pipes, an electrician check the wires, and a carpenter check the frame. If the frame is crooked, the carpenter gives a specific "bad score" so the builder knows to fix the frame, not the paint.

4. The Result

By using this method, the robot became much better at reading complex documents.

They tested it on a famous benchmark called OmniDocBench (which has 1,355 pages of tricky documents).
The robot scored 90.41, beating all other "end-to-end" models (models that try to do everything in one go).
It was particularly good at fixing the "confusion" in formulas and tables, proving that teaching the robot to reason about structure works better than just making it memorize more text.

Summary

The paper argues that reading complex documents isn't just about "seeing" text; it's about reasoning through the structure. By filtering out easy examples and giving the robot specific, separate feedback for text, math, and tables, they taught it to stop guessing and start understanding the rules of the document.

Technical Summary: Format Decoupled Reinforcement Learning for Document OCR

1. Problem Statement

While Optical Character Recognition (OCR) has long been a focus of research, existing advanced models still exhibit significant limitations when processing format-sensitive document content. The authors observe that while text reading is often perceived as a perceptual task, advanced OCR models struggle with high output uncertainty (entropy) when dealing with structured elements like formulas and tables compared to plain text.

Empirical analysis reveals that document samples with a higher proportion of formatted text exhibit substantially higher token entropy during inference. This high entropy indicates lower model certainty and suggests that formatted text allows for diverse, semantically equivalent reading pathways (e.g., different LaTeX representations for the same formula). Existing Reinforcement Learning (RL) attempts in OCR often lack specific design considerations for these high-entropy formatted regions, failing to move beyond token-level memorization toward format-level reasoning.

2. Methodology

The paper proposes Format Decoupled Reinforcement Learning (FD-RL), a framework designed to optimize OCR performance by explicitly targeting format-sensitive content through a two-stage training paradigm: Supervised Fine-Tuning (SFT) followed by RL.

2.1 Data Engineering

To support robust training, the authors constructed a large-scale, format-rich corpus integrating three distinct sources:

Open-Source Dataset Enhancement: Datasets (e.g., PDFA, DocGenome) were quality-checked for missing content, incorrect reading orders, and repetitions. A lightweight model (GOT) was used for re-annotation, and similarity-based filtering ensured high-quality ground truth retention.
Real-World PDF Construction: Diverse PDFs were processed to create layout-aware (page-level, region-level, multi-page) and content-aware annotations. Tools like MinerU and dots.ocr were utilized to generate structured Markdown, LaTeX, and HTML outputs.
Synthetic OCR Data Generation: To address the scarcity of educational and academic data, synthetic samples were generated from online resources (K12 to graduate STEM Q&A). These were rendered into high-resolution images with realistic visual parameters (fonts, spacing) and paired with Markdown labels.

2.2 Training Strategy

The training pipeline consists of two distinct stages:

Stage 1: Supervised Fine-Tuning (SFT)
A pre-trained Vision-Language Model (Qwen3-VL-4B) is fine-tuned on the comprehensive dataset. During this phase, the visual encoder and projector are frozen, while only the Large Language Model (LLM) parameters are updated. This establishes a strong baseline for general OCR capabilities across diverse layouts and formats.

Stage 2: Format Decoupled Reinforcement Learning (FD-RL)
The RL stage introduces two core innovations to shift optimization from token-level accuracy to format-level validity:

Entropy-Based Data Filtration:
Instead of using all SFT data, the authors filter the dataset to retain high-entropy samples. The SFT model performs inference on candidate samples, and the average token entropy is calculated. Samples exceeding a specific entropy threshold ( $\tau$ ) are selected for RL training. This strategy concentrates learning on complex, format-intensive instances where the model exhibits the most uncertainty.
Format Decoupled Reward:
Rather than applying a uniform reward signal, the framework decouples rewards based on content type:
- Plain Text: Evaluated using normalized string matching (Edit Distance).
- Formulas: Evaluated using expression correctness (converting to equivalent LaTeX and applying BLEU scores).
- Tables: Evaluated using structural coherence (TEDS score).
The total reward $R$ is a weighted sum of these specific rewards, ensuring that structural errors (e.g., broken table hierarchies or malformed LaTeX) are not drowned out by content errors. The model is optimized using the GRPO algorithm, which utilizes group-normalized advantages without requiring a separate critic model.

3. Key Contributions

FD-RL Framework: Proposes a novel SFT-then-RL paradigm specifically designed for "reasoning-after-reading" in document OCR, addressing the high-entropy nature of formatted text.
Large-Scale Format-Rich Corpus: Constructs a robust training dataset by integrating open-source, real-world, and synthetic data with optimized mixing ratios.
Entropy-Based Curation & Decoupled Rewards: Introduces a data filtration strategy that targets high-uncertainty samples and a reward mechanism that tailors evaluation metrics to specific content formats (text, formula, table).
State-of-the-Art Performance: Achieves a new record on the OmniDocBench benchmark, validating the effectiveness of the proposed approach through comprehensive ablation studies.

4. Experimental Results

The method was evaluated on OmniDocBench, a benchmark containing 1,355 pages across nine document types (books, notes, slides, etc.).

Overall Performance: FD-RL achieved an overall score of 90.41, setting a new record for end-to-end models. It outperformed the previous best end-to-end model (dots.ocr) by 2.0 points and the best pipeline-based model (PaddleOCR-VL) by a significant margin in specific metrics, though PaddleOCR-VL scored 92.56 overall (note: PaddleOCR-VL is listed as pipeline-based in Table 1, while FD-RL is end-to-end).
Specific Metrics:
- Formula Recognition: Achieved the best CDM score of 88.67.
- Table Parsing: Achieved a top TEDS score of 87.35 and a TEDS-S score of 92.10.
- Text Recognition: Achieved an edit distance of 0.049, ranking second only to dots.ocr (0.048).
Generalization: FD-RL secured the top performance in 4 out of 9 document categories (e.g., Slides, Exam Papers) and ranked second in the remaining 5, demonstrating strong cross-domain generalization.

Ablation Studies:

Data Sources: Progressive integration of open-source, real-world, and synthetic data, followed by RL, showed consistent performance gains, with the full pipeline yielding the highest score.
Training Strategy: Applying RL directly to a general VLM without SFT yielded minimal gains (49.37), whereas SFT alone achieved 87.13. The combination of SFT+RL was necessary to reach 90.41.
Filtration: A 50% entropy-based filtration rate yielded optimal results; both 0% (no filtration) and 75% (excessive filtration) led to performance degradation.
Reward Design: The full format-decoupled reward scheme outperformed unified string-matching rewards, confirming that specialized evaluation for formulas and tables is critical.

5. Significance and Claims

The paper claims that the primary significance of this work lies in identifying and leveraging the high-entropy nature of formatted text as a signal for targeted optimization. By observing that advanced models struggle with format-sensitive content due to diverse valid structural expressions, the authors argue that reasoning over diverse reading pathways is essential for improving OCR.

The authors assert that FD-RL successfully shifts the optimization objective from simple token-level memorization to format-level validity. This is evidenced by the model's ability to generate correct HTML structures for tables and valid LaTeX for formulas, tasks where baseline models often fail by producing plain text or instruction-following errors. The work validates that entropy-based data curation and format-specific reward design are effective strategies for enhancing the robustness of end-to-end document parsing systems.