💻 computer science

An LMM for Precisely Grounding Elements in Documents

The paper introduces PreciseDoc, a Large Multimodal Model designed to improve visual grounding precision in text-rich documents through the use of mass-produced synthetic training data and a joint reinforcement learning paradigm that unifies grounding with reasoning.

Original authors: Yijian Lu, Chuangxin Zhao, Kai Sun, Lei Hou, Juanzi Li, Ji Qi

Published 2026-06-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yijian Lu, Chuangxin Zhao, Kai Sun, Lei Hou, Juanzi Li, Ji Qi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, messy library of documents—resumes, academic papers, charts, and forms. You ask a smart robot (a Large Multimodal Model, or LMM) to find a specific piece of information, like "Show me the phone number on this resume."

Current robots are like students who can read the text but are terrible at pointing. They might say, "The phone number is 555-0199," but they can't tell you where on the page that number is. If you asked them to draw a box around it, they might draw it over the wrong paragraph, or miss it entirely. This is a problem because if the robot can't point to the evidence, you can't trust its answer.

The paper "PreciseDoc" introduces a new robot designed specifically to be a master "pointer" and a better thinker. Here is how they built it, explained simply:

1. Building a Better Training Gym (The Data Engine)

To teach a robot to point accurately, you need practice materials. The authors realized that existing practice materials were either too easy or didn't have the "answer key" (the exact coordinates) needed for training.

So, they built two factories to mass-produce perfect practice documents:

The "Digital Blueprint" Factory: They used computer code (LaTeX) to generate thousands of perfect resumes and documents. Because they built these from code, they knew exactly where every word was located, like having a digital blueprint of a house.
The "Handwritten Note" Factory: Real documents often have messy handwriting or look like they were scanned with a shaky camera. To teach the robot to handle this, they created a system that assembles individual handwritten letters (like building blocks) to form sentences, then adds "camera effects" like blur or shadows. This creates synthetic documents that look and feel like real-world messiness but still come with a perfect "answer key."

2. The Two-Step Training Process

The robot didn't just learn to point; it learned to think while pointing. The training happened in two stages:

Stage 1: The "Cold Start" (Learning the Ropes)
Before the robot could learn on its own, the researchers gave it a "cheat sheet." They took existing questions and answers and manually inserted the correct "pointing boxes" into the robot's thought process.

Analogy: Imagine teaching a child to solve a math problem by showing them the steps and highlighting exactly which numbers on the page they are using for each step. The robot learned to say, "I am looking at this box here to find the answer," before giving the final result.

Stage 2: Reinforcement Learning (The Coach with a Whistle)
Once the robot knew the basics, they let it practice on its own, but with a strict coach.

The Reward System: When the robot guessed an answer, the coach checked two things:
1. Did you get the right answer? (Answer Reward)
2. Did you point to the right spot? (Grounding Reward)
The "Hungarian Algorithm" Trick: This is a fancy math tool the researchers used to match the robot's boxes to the real boxes perfectly.
- The Problem: If a robot draws 10 boxes that are all slightly different but cover the same word, a lazy scoring system might give it full credit because "it found the word."
- The Fix: The Hungarian Algorithm forces a one-to-one match. If the robot draws 10 boxes for one word, only one gets credit, and the other 9 get punished. This stops the robot from "spamming" boxes to trick the system.
The Length Penalty: The coach also penalized the robot if it drew too many boxes that didn't match anything. This forced the robot to be precise and not just guess wildly.

3. The Results

The new robot, called PreciseDoc (and its reasoning version, PreciseDoc-Reasoner), was tested against other top-tier robots.

Pointing: It became much better at drawing tight, accurate boxes around words, phrases, and lines in documents, outperforming many existing models.
Thinking: When asked to solve complex document questions, it didn't just guess; it pointed to the specific evidence it used to reach the conclusion. It was able to locate personal info in resumes or data in charts with high precision.
General Understanding: Even though it was trained specifically to point and reason, it remained good at general document understanding tasks, performing competitively with much larger, more complex models.

Summary

In short, the authors built a robot that doesn't just "read" documents but can "see" and "point" to exactly where information lives. They did this by creating a massive library of synthetic practice documents with perfect answer keys and teaching the robot a strict rule: You must point to the evidence to prove your answer is right. This makes the robot's reasoning transparent and much more reliable.

Technical Summary: An LMM for Precisely Grounding Elements in Documents

Problem Statement

Visual grounding—the ability of Large Multimodal Models (LMMs) to locate specific visual regions based on natural language instructions—is critical for document understanding, deep research, and error detection. However, existing LMMs struggle with precise grounding in text-rich document images. Unlike natural images, documents present high element density, rich template diversity, and complex layouts. Current models often fail to accurately locate critical elements (words, phrases, lines), leading to "hallucinated" reasoning where the model focuses on unrelated regions. This deficiency hinders visual grounded reasoning, a process where an LMM must cite specific spatial coordinates as evidence for its textual reasoning. Furthermore, existing datasets lack the granularity and diversity (e.g., synthetic hand-filled documents) required to train models for these challenging scenarios, and no complete training pipeline exists that jointly optimizes grounding and reasoning via reinforcement learning (RL).

Methodology

The paper introduces PreciseDoc, an LMM designed for precise element grounding, and PreciseDoc-Reasoner, its optimized variant for visual grounded reasoning. The methodology is structured into three core components:

1. High-Quality Data Generation Pipelines

To address the scarcity of high-quality grounding data, the authors constructed two automated document generation engines:

LaTeX-based Engine: Compiles LaTeX source codes to generate native PDFs. A custom command (\marktext) is inserted during compilation to capture precise spatial coordinates (converted from points to pixels) for wrapped text. This engine supports mass production of diverse resume templates and other document types.
Handwritten Data Engine: Synthesizes hand-filled documents with camera effects (e.g., filters, degradation) to simulate real-world scanning. It assembles individual character samples from the NIST dataset to form coherent text, reserving spaces for values generated by a VLM. This pipeline addresses the lack of datasets for handwritten document grounding.
Task Construction: These engines generate paired documents and metadata for two tasks: Pure Grounding (locating specific phrases or elements) and Information Extraction (locating groups of elements, such as "Work Experience" in a CV).

2. Training Pipeline: From Cold-Start to Reinforcement Learning

The training of PreciseDoc-Reasoner follows a two-stage paradigm:

Stage 1: Cold-Start Initialization (SFT): The model is fine-tuned on a curated 27K VQA dataset (derived from BBox-DocVQA and DOGR). This dataset includes reasoning trajectories where the model is instructed to generate bounding box coordinates (at either token-level or element-level) alongside its thought process and final answer. The data is constructed by expanding token-level boxes to element-level boxes and annotating content to ensure the VLM actively uses the boxes for reasoning rather than treating them as supplementary tags.
Stage 2: Reinforcement Learning (RL): The model undergoes joint RL training to supervise both reasoning accuracy and intermediate grounding.
- Reward Design: The total reward $R$ combines an answer accuracy reward ( $R_{acc}$ ) and a dual Intersection-over-Union (IoU) reward ( $R_{IoU}$ ).
- Hungarian Algorithm for Matching: Unlike previous methods using many-to-many mapping (which can inflate metrics), the authors employ the Hungarian Algorithm to establish an optimal one-to-one assignment between predicted and ground-truth boxes. This maximizes the total IoU while ensuring strict mapping constraints.
- Length Penalty: A strict penalty is applied to unmatched generated boxes to prevent the model from producing redundant or excessive bounding boxes to "game" the IoU score.

Key Contributions

Document Generation Engines: The introduction of two pipelines capable of mass-producing high-quality documents with paired, fine-grained spatial metadata, including synthetic hand-filled documents with camera effects.
PreciseDoc Model: An LMM achieving State-of-the-Art (SOTA) performance in document text grounding, capable of real-world functions like locating personal information in CVs, surpassing existing specialized models.
PreciseDoc-Reasoner & Training Paradigm: The first complete training pipeline (SFT to RL) for visual grounded reasoning in documents. It introduces a novel RL reward mechanism using the Hungarian Algorithm for one-to-one box matching and a length penalty to ensure precise, non-redundant evidence localization.

Experimental Results

The proposed models were evaluated on multiple benchmarks:

DocLocal4K (Text Grounding): PreciseDoc achieved an overall accuracy of 81.79%, outperforming strong baselines like Doubao-Seed-2.0-Pro (54.45%) and DocOwl-1.5 (80.38%). It demonstrated superior performance on small-granularity elements (words, phrases, lines).
TRIG (Word-Level Grounded Reasoning): PreciseDoc-Reasoner achieved an average IoU of 46.42%, significantly outperforming all other open-source and proprietary models (e.g., Qwen3.5-27B at 20.03%, Gemini-3-Pro at 30.26%). It showed particular strength in chart understanding, doubling the performance of the second-best model.
BBox-DocVQA (Element-Level Grounded Reasoning): The model achieved a grounding accuracy of 51.16% and answer accuracy of 75.97%. It outperformed all listed open-source LMMs (including the 235B parameter Qwen3VL) and ranked second only to Gemini-3-Flash among proprietary models.
Traditional Benchmarks: On DocVQA, ChartQA, and TextVQA, PreciseDoc-Reasoner demonstrated competitive performance, achieving results comparable to or better than larger models like Qwen3.5-9B, despite not being trained on massive general VQA datasets.

Significance and Claims

The paper claims that PreciseDoc successfully bridges the gap between document understanding and precise spatial localization. By constructing challenging, high-fidelity training data and introducing a rigorous RL training paradigm with one-to-one box matching, the authors demonstrate that LMMs can be effectively trained to provide verifiable visual evidence. The work highlights that accurate grounding is not merely a preprocessing step but a critical component of the reasoning process that prevents hallucination and enhances interpretability. The authors position their approach as a foundational step toward more reliable, evidence-based document analysis systems, noting that the proposed methods and data generation pipelines offer a scalable solution for future advancements in document AI.