DianJin-OCR-R1: Enhancing OCR Capabilities via a Reasoning-and-Tool Interleaved Vision-Language Model

DianJin-OCR-R1 is a reasoning-enhanced vision-language model that improves OCR accuracy by interleaving its own recognition with expert tool outputs, guiding the model to iteratively re-examine images and integrate evidence to reduce hallucinations and enhance fine-grained perception.

The Gist

, , `), it gets points.
* If the final answer is correct, it gets huge points.
* If it hallucinates or skips the "re-check" step, it gets zero points.
* Result: The AI learns that slowing down and double-checking is the only way to win.

Why Is This a Big Deal?

1. It Fixes "Hallucinations": By forcing the AI to compare its guess with hard data from other tools, it stops making up words that aren't there.
2. It's Cheaper and Faster: Usually, to make an AI smarter, you have to retrain the whole massive brain (which costs millions of dollars). Here, if a new, better "Specialist Tool" comes out, you just swap the tool. The main AI doesn't need to be retrained; it just learns to use the new tool better.
3. It Actually "Looks": The researchers proved that during the "Rethink" phase, the AI's attention mechanism literally spikes, focusing back on the image pixels. It's not just guessing; it's genuinely re-examining the evidence.

The Bottom Line

DianJin-OCR-R1 is like teaching an AI to be a meticulous editor rather than a fast typist. Instead of rushing to type a document, it drafts, consults a dictionary and a grammar checker, re-reads the original text, and then types the final version. The result? Documents that are read with human-like understanding but machine-like precision.

Technical Summary

):** The VLM attempts to recognize the content in the image using its own internal OCR capabilities. 2. **Tool Invocation ():** The model calls external expert models (e.g., PP-StructureV3, MonkeyOCR, GOT) to obtain reference results for the same image. 3. **Re-evaluation ():** Guided by the tool results, the VLM is instructed to "look again" at the image. It compares its initial recognition with the tool outputs to identify errors, omissions, or ambiguities. This step implicitly forces the model to re-focus its attention on the visual input. 4. **Final Integration (`):** The model synthesizes all available evidence (its own observation + tool references) to generate the final, most accurate output.

B. Data Construction

To train this paradigm, the authors constructed reasoning datasets for three specific tasks: Seal Recognition, Table Recognition, and Formula Recognition.

• Process: They used powerful VLMs (like Qwen-VL-Max) to generate interleaved reasoning chains.
• Filtering: A sample is retained only if the final output matches the ground truth (or exceeds a specific similarity threshold like TEDS for tables or NED for formulas) and the reasoning chain correctly utilizes the tool results to correct initial errors.
• Datasets: Used ReST (Seals), in-house/TabRecSet (Tables), and UniMER-1M (Formulas).

C. Training Strategy

• Supervised Fine-Tuning (SFT): The model learns to generate the CoT (Chain-of-Thought) structure (<think>, <tool>, <rethink>, <answer>) based on the constructed datasets.
• Reinforcement Fine-Tuning (RFT): The model is further optimized using Group Relative Policy Optimization (GRPO). Two reward signals are used:
  • Format Reward: Ensures the output strictly adheres to the required XML-like tags.
  • Accuracy Reward: Rewards correct answers (exact match for seals, TEDS for tables, CDM/NED for formulas).

3. Key Contributions

1. Novel Framework: Introduced DianJin-OCR-R1, a reasoning-enhanced framework that effectively balances the semantic understanding of VLMs with the recognition precision of expert OCR models via a tool-interleaved approach.
2. Implicit "Look-Again" Mechanism: Demonstrated that the reasoning process guides VLMs to implicitly re-focus on visual inputs, significantly reducing hallucinations.
3. Cost-Efficient Iteration: The method allows performance improvements simply by upgrading the external "expert tools" without requiring expensive re-training of the base VLM.
4. Synthetic Data Generation: The framework can generate high-quality synthetic reasoning data using advanced models, further advancing document parsing research.

4. Experimental Results

The model was evaluated on ReST and OmniDocBench benchmarks across Seal, Table, and Formula recognition tasks.

• Performance vs. Baselines: DianJin-OCR-R1 consistently outperformed:
  • General VLMs (e.g., Qwen2.5-VL-7B, InternVL3): Achieving significant gains in accuracy (e.g., Seal ACC improved from 0.527 to 0.766).
  • Expert VLMs (e.g., GOT, MonkeyOCR): Surpassing models specifically trained for OCR.
  • Expert OCR Models (e.g., PP-StructureV3): Even outperforming the specialized tools used within the reasoning chain.
• Ablation Studies:
  • Tool Usage: Providing tool results to the base model improved performance, but the full reasoning framework (SFT/RFT) yielded the best results, proving the value of the "rethink" step.
  • RFT vs. SFT: Models trained with Reinforcement Fine-Tuning (RFT) generally outperformed those trained only with SFT, particularly in the "look-again" attention mechanism.
• Attention Analysis: Visualization of attention maps confirmed that during the <rethink> stage, the model's attention spikes on image tokens, validating that the model is genuinely re-examining the visual input rather than just relying on text priors.

5. Significance

DianJin-OCR-R1 represents a paradigm shift in OCR tasks by moving away from isolated recognition models toward collaborative, reasoning-driven systems.

• Reduced Hallucination: By forcing the model to verify its own predictions against expert tools and re-examine the image, it significantly mitigates the "imagining words" problem common in VLMs.
• Scalability: The approach decouples the reasoning engine (VLM) from the recognition engine (Tools). As better OCR tools are developed, the system's performance automatically improves without the need for costly VLM re-training.
• Transparency: The structured reasoning process (<think> -> <tool> -> <rethink>) makes the model's decision-making process transparent and verifiable, which is crucial for high-stakes document processing applications like RAG (Retrieval-Augmented Generation).