💬 NLP

Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA

Doc-V* is an OCR-free agentic framework for multi-page Document VQA that employs a coarse-to-fine interactive reasoning strategy, combining thumbnail overviews with active semantic retrieval and structured memory aggregation to achieve superior accuracy and efficiency compared to existing baselines.

Original authors: Yuanlei Zheng, Pei Fu, Hang Li, Ziyang Wang, Yuyi Zhang, Wenyu Ruan, Xiaojin Zhang, Zhongyu Wei, Zhenbo Luo, Jian Luan, Wei Chen, Xiang Bai

Published 2026-04-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuanlei Zheng, Pei Fu, Hang Li, Ziyang Wang, Yuyi Zhang, Wenyu Ruan, Xiaojin Zhang, Zhongyu Wei, Zhenbo Luo, Jian Luan, Wei Chen, Xiang Bai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are handed a massive, 200-page financial report and asked a single, specific question: "How many signatures are on page 7?"

The Old Way (The "Brute Force" Approach):
Most current AI models try to read the entire 200-page document at once, like a student trying to swallow a whole textbook in one bite.

The Problem: The AI gets overwhelmed. It forgets the beginning by the time it reaches the end (the "lost in the middle" effect). Or, it tries to find the answer by blindly guessing which 5 pages might be important, often picking the wrong ones and missing the signature entirely.
The Result: It's either too slow, too expensive, or just plain wrong.

The New Way (Doc-V): The "Detective" Approach*
The paper introduces Doc-V*, which acts less like a robot reading a book and more like a human detective solving a mystery. Instead of reading everything, it uses a smart, step-by-step strategy called "Coarse-to-Fine Interactive Visual Reasoning."

Here is how Doc-V* works, using a simple analogy:

1. The "Bird's-Eye View" (The Thumbnail)

Before the detective dives into the evidence, they look at a polaroid collage of the whole document.

What happens: The AI looks at tiny, low-resolution thumbnails of every page. It can't read the small text, but it can see the shape of things.
The Analogy: It's like flipping through a photo album quickly. You don't read the captions yet, but you see, "Oh, page 7 has a big chart," or "Page 12 looks like a contract with signatures." This gives the AI a map of where to look.

2. The "Active Search" (The Detective's Tools)

Instead of waiting for the answer to be handed to it, Doc-V* actively goes hunting. It has two main tools:

The "Ctrl+F" Tool (Retrieval): If the question is "Find the signature," the AI acts like a search engine, scanning the document images for keywords or visual patterns to find the right pages.
The "Grab the File" Tool (Fetch): If the AI sees a signature on the thumbnail of page 7, it doesn't just guess; it specifically says, "I need the high-resolution version of page 7 right now." It fetches only that page to read the fine print.

3. The "Working Memory" (The Detective's Notebook)

As the detective finds clues, they write them down in a notebook so they don't forget.

What happens: Doc-V* keeps a running summary of what it has found. If it finds a signature on page 7, it writes it down. If it looks at page 8 and finds nothing, it notes that too.
The Benefit: This prevents the AI from getting confused or looping in circles. It builds a coherent story from scattered clues.

4. The "Stop When Done" Rule

A human detective doesn't keep searching after they've found the killer. Doc-V* is trained to know when it has enough evidence.

The Training: The AI was taught by watching an expert (GPT-4o) solve these problems. Then, it played a game where it got "points" for finding the right answer quickly and accurately. If it wasted time reading irrelevant pages, it lost points. This taught it to be efficient.

Why is this a big deal?

It's Smarter: It doesn't just throw more computing power at the problem; it changes how it looks. It focuses on quality of evidence rather than quantity of pages.
It's Cheaper: By only reading the pages it actually needs, it saves massive amounts of computer memory and time.
It's Robust: Even if the document is messy or the question is tricky, Doc-V* adapts. If it can't find the answer on page 7, it doesn't give up; it decides to check page 8 or search for a different keyword.

In a nutshell:
Old AI models are like a person trying to find a specific word in a library by reading every single book cover-to-cover. Doc-V* is like a librarian who knows exactly which shelf to walk to, pulls out the specific book, flips to the right page, and gives you the answer in seconds. It turns document reading from a passive chore into an active, intelligent investigation.

(thought) andprotocol. * **Thinking Block:** Includes(content inspection),(subgoal decomposition), and

(incremental update).     *   **Working Memory:** The agent maintains a concatenated summary of previous turns to prevent forgetting and drift during long interactions.     *   **Relevance Judgment:** The agent explicitly lists

` in its thought process, forcing binary relevance decisions.

B. Training Strategy

Doc-V ∗ employs a two-stage optimization pipeline:

Supervised Fine-Tuning (SFT):
- Distills high-quality interaction trajectories from a strong teacher model (GPT-4o) interacting with the real environment.
- Trajectories are filtered for format validity, answer correctness, and evidence sanity.
- Loss is computed only on agent-generated tokens (thoughts and actions), not environment observations.
Group Relative Policy Optimization (GRPO):
- A reinforcement learning phase that optimizes for answer accuracy and evidence-seeking efficiency simultaneously.
- Uses a composite reward function: $R = \omega_{ans}R_{ans} + \omega_{evi}R_{evi} + \omega_{struct}R_{struct}$ .
- Optimizes the policy by sampling groups of trajectories and maximizing the relative advantage within the group, encouraging the agent to find answers with fewer, more relevant pages.

3. Key Contributions

Agentic Framework for Long Documents: Introduces a novel "coarse-to-fine" paradigm where an OCR-free agent actively navigates documents via thumbnails, semantic retrieval, and targeted fetching, rather than passively ingesting fixed contexts.
Structured Working Memory: Implements a mechanism to maintain a summary of visited pages and reasoning steps, mitigating the "lost-in-the-middle" effect and enabling multi-hop reasoning over long contexts.
Efficient Evidence Aggregation: Demonstrates that high performance comes from selective attention and evidence consolidation, not merely increasing the number of input pages.
Robust Training Pipeline: Combines SFT with GRPO to balance answer quality with the cost of evidence retrieval, allowing small open-source models to rival proprietary systems.

4. Experimental Results

The model was evaluated on five benchmarks: DUDE, MP-DocVQA, SlideVQA, MMLongBench-Doc, and LongDocURL.

Performance:
- Out-of-Domain (OOD): Doc-V ∗ achieves a 47.9% improvement over static RAG baselines on OOD tasks.
- Comparison: It outperforms existing open-source baselines (including E2E and RAG models) and approaches the performance of proprietary models like GPT-4o and Claude-3.7-Sonnet.
- Specific Gains: On LongDocURL, it improves accuracy by +23.4% over the base Qwen2.5-VL and +18.5% over a RAG Top-5 baseline.
Efficiency & Robustness:
- Page Selection: Doc-V ∗ achieves significantly higher Page-F1 (alignment with ground-truth evidence pages) than RAG, proving it selects better pages, not just more pages.
- Document Length: Performance remains stable even for documents >80 pages, whereas E2E and RAG methods degrade significantly.
- Retriever Independence: The framework remains robust even when paired with weaker text-based retrievers (e.g., BM25), as the agent can actively compensate for retrieval misses via browsing actions.
Ablation Studies:
- Removing the Global Thumbnail or Page-by-Page Analysis modules causes significant performance drops (4.9 and 4.7 accuracy points, respectively), confirming the necessity of multi-granularity perception.
- Both Retrieval and Fetch actions are complementary; removing either degrades performance.

5. Significance

Doc-V ∗ represents a paradigm shift in long-document understanding. It moves away from the "brute force" approach of feeding entire documents into models or relying on brittle static retrieval. Instead, it establishes active, goal-directed perception as the key to solving complex DocVQA tasks.

By demonstrating that selective evidence aggregation is more effective than sheer context volume, the paper provides a scalable, cost-effective solution for processing long, visually rich documents. This approach allows smaller, open-source models to achieve state-of-the-art results, reducing reliance on massive proprietary models while maintaining high robustness across diverse document types and domains.