💻 computer science

DocClaw: A Unified Agentic System for Intelligent Document Processing

DocClaw is a unified agentic system that reframes diverse intelligent document processing tasks as a shared iterative interaction process between an agent and a document, leveraging a structured document state and reusable tools to achieve competitive performance across OCR, DocQA, and KIE benchmarks.

Original authors: Siqi Xiang, Zhipeng Xu, Yufei Liu, Junhao Ji, Qing Liu, Zulong Chen, Zhibo Yang, Chunyan Miao, Shijian Lu

Published 2026-08-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Siqi Xiang, Zhipeng Xu, Yufei Liu, Junhao Ji, Qing Liu, Zulong Chen, Zhibo Yang, Chunyan Miao, Shijian Lu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Every day, we are surrounded by documents that hold the world's knowledge: annual reports, medical records, legal contracts, and handwritten notes. For a computer to understand these pages, it must do more than simply read the words; it must see how the text is arranged, recognize charts and tables, and piece together information that is scattered across different sections. This challenge, known as intelligent document processing, has long been treated as a collection of separate jobs. One program might be designed to turn a picture of a page into text, while a different program is needed to find a specific answer within that text, and yet another to pull out key numbers like dates or prices. These tools work in isolation, unable to share what they learn from one task to help with the next, forcing a rigid, step-by-step approach that often misses the bigger picture.

Researchers have now introduced a new system called DocClaw that changes how computers interact with documents. Instead of treating each task as a separate problem, DocClaw acts like a single, intelligent assistant that can read, search, and reason through a document in a continuous loop. When given a question or a request, this system does not just guess an answer; it actively explores the document, using a set of specialized tools to find the right information, check its work, and refine its understanding until it is confident in the result. By keeping a running record of what it has already discovered, the system can reuse that knowledge for future questions, making it faster and more accurate over time.

The core of this new approach is a shift from static prediction to active interaction. In traditional methods, a computer looks at a document once and produces an output, with no way to go back and correct a mistake or look deeper if the first attempt was unclear. DocClaw, however, treats the document as a partner in a conversation. When a user asks a question, the system first decides on a strategy based on what is being asked. If the goal is to simply read the text, it focuses on recognizing letters and layout. If the goal is to find a specific fact, it plans a search. If the goal is to extract a value like a revenue number, it prepares to cross-check its findings. This strategy is guided by what the researchers call "document skills," which are essentially instructions that tell the system how to behave for different types of tasks.

Once the strategy is set, the system begins its work by building a structured memory of the document. It breaks the page down into manageable parts, noting where text, images, and tables are located. As it uses its tools to read specific sections or analyze a chart, it saves those findings in a persistent memory bank. This is a crucial difference from older systems: the knowledge gained from reading one part of a report is not discarded after the task is done. Instead, it is stored so that if a later question requires information from that same section, the system already knows where to look and what it found. This allows the system to avoid repeating work and to build a deeper understanding of the document as it answers more questions.

The system operates in a cycle of planning, acting, and updating. It looks at the current state of its knowledge, decides what tool to use next—such as zooming in on a blurry section, cropping a table for better clarity, or searching for a keyword—and then executes that action. The result of that action is immediately added to its memory. If the system finds a number that seems inconsistent, it can use its tools to verify it against the original image or run a second check. This process continues until the system has gathered enough evidence to answer the question with confidence. Only then does it stop and provide the final answer, discarding the temporary notes for that specific question while keeping the permanent knowledge of the document for future use.

To test this approach, the researchers evaluated DocClaw on three very different types of tasks: reading text from images, answering questions about long documents, and extracting specific data points like financial figures. They compared the system against both general-purpose artificial intelligence models and specialized tools designed for single tasks. The results showed that DocClaw performed as well as, or better than, the specialized tools across the board. In tests involving the recognition of complex text, formulas, and tables, the system achieved high accuracy scores, often outperforming dedicated software. When asked to answer questions based on long reports, it was able to locate the correct information and provide grounded answers more reliably than standard models. Similarly, when tasked with pulling out key information, it successfully combined visual reading with text recognition to reduce errors.

A closer look at how the system works revealed why it was so effective. The researchers found that the ability to refine its own work was a major factor. When the system encountered difficult-to-read text, it frequently used a "zoom" tool to get a closer look, which significantly improved its ability to recognize small symbols and numbers. In tasks requiring the extraction of specific data, the system's habit of double-checking its findings against the original image corrected a large portion of the mistakes that other models made. Furthermore, the system's memory proved essential for efficiency. When asked a series of questions about the same document, the time it took to answer each subsequent question dropped noticeably. This happened because the system did not need to re-scan the entire document for every new query; it could simply recall what it had already learned and focus only on the new information needed.

The study demonstrates that treating document processing as a unified, interactive process is a powerful way to handle the complexity of real-world documents. By allowing a single system to switch between reading, searching, and verifying based on the needs of the moment, DocClaw overcomes the limitations of rigid, single-purpose tools. It shows that when a computer is given the ability to remember what it has seen and to plan its next move carefully, it can understand documents with a level of flexibility and accuracy that was previously difficult to achieve. This approach does not just automate a task; it creates a more intelligent way for machines to engage with the vast amount of written information that shapes our world.

Technical Summary: DocClaw

Problem Statement

Intelligent Document Processing (IDP) encompasses diverse tasks such as Optical Character Recognition (OCR), Document Question Answering (DocQA), and Key Information Extraction (KIE). While these tasks share a fundamental need to perceive document content and acquire relevant information, existing approaches typically treat them as isolated prediction problems. Current methods generally fall into two paradigms:

Single-Pass Prediction: Models directly output task-specific results without intermediate steps (e.g., standard OCR or end-to-end VLMs). These rely on rigid, task-specific mappings and lack flexibility.
Multi-Step Task-Specific Pipelines: These decompose processing into predefined sequences of specialized operations (e.g., layout analysis followed by region recognition). While effective for specific tasks, these pipelines are constrained by their predefined structures and cannot easily adapt to heterogeneous objectives.

The primary limitations of these existing paradigms are the lack of a unified framework for handling diverse tasks within a single workflow and the inability to accumulate, reuse, or refine document knowledge across different queries. Consequently, intermediate results from one operation cannot serve as reusable knowledge for subsequent tasks, leading to redundant processing and isolated knowledge silos.

Methodology: DocClaw

To address these limitations, the authors introduce DocClaw, a unified agentic system that reformulates IDP as an iterative interaction between an agent and a document. The system operates through three core components:

1. Agentic Interaction Loop

DocClaw treats document processing as a sequence of interactions rather than a single forward pass. Given a document $D$ and a query $q$ :

Skill Selection: The agent first selects a Document Skill ( $k$ ) from a predefined set based on the query objective. This skill defines the interaction strategy and tool-use patterns.
State Initialization: A structured Document State ( $s_0$ ) is initialized, comprising Document Memory (persistent knowledge) and Task Memory (query-specific context).
Iterative Planning and Execution: Using a ReAct-style loop, the agent reads a condensed representation of the current state, the query, and the selected skill to determine the next action ( $a_t$ ). This action involves selecting a document processing tool ( $\tau_t$ ) and its arguments ( $c_t$ ).
State Update: The tool execution produces a result ( $r_t$ ), which is integrated into the document state. The state update function separates reusable document knowledge (accumulated in Document Memory) from task-specific context (tracked in Task Memory).
Termination: The loop continues until the agent determines sufficient information has been gathered to generate the final output. Upon completion, the Task Memory is discarded, while Document Memory is retained for future queries on the same document.

2. Structured Document State

The system maintains a hierarchical state $s_t = (m_t, e_t)$ to organize information:

Document Memory ( $m_t$ ): A persistent, hierarchical structure (document $\to$ page $\to$ region) that stores raw images, recognized text, layout structures, and tool-produced attributes. This allows knowledge acquired by one tool (e.g., layout analysis) to be refined and reused by others (e.g., text recognition) across different queries.
Task Memory ( $e_t$ ): A query-level context that records the selected skill, interaction history, inspected regions, and intermediate evidence. This ensures the agent has the necessary context for planning but prevents irrelevant task-specific noise from affecting future queries.

3. Document Skills

To guide the agent's strategy across different objectives, DocClaw employs task-oriented Document Skills:

OCR-Oriented Skill: Focuses on accurate transcription. It guides layout analysis to route regions to specific parsers (text, formula, table) and encourages visual enhancement (zoom, crop, rotate) for low-quality regions before re-recognition.
DocQA-Oriented Skills: Divided into Inspection (coarse-to-fine search for localized evidence) and Enumeration (broad coverage for document-wide reasoning) to handle varying question types.
KIE-Oriented Skill: Targets field extraction by combining visual extraction, OCR-based extraction, and image-grounded verification to resolve conflicts and improve accuracy.

Key Contributions

The paper claims three main contributions:

Unified Agentic Framework: DocClaw formulates heterogeneous IDP tasks as iterative agent-document interactions, allowing diverse query objectives to be addressed within a shared interaction loop and tool space.
Structured Document State: The proposal of a dual-memory system (Document Memory and Task Memory) that supports the progressive accumulation, reuse, and refinement of document knowledge across interactions, while isolating task-specific context.
Document Skills: The introduction of a set of task-oriented instructions that guide the agent's processing strategies and tool-use patterns, enabling dedicated refinement operations tailored to specific query objectives.

Experimental Results

The authors evaluated DocClaw on three benchmarks: OmniDocBench v1.6 (OCR), MMLongBench-Doc (DocQA), and OCRBench/OCRBench v2 (KIE). The system was tested using three backbone models: GPT-5.2, Gemini-2.5-Flash, and Gemini-2.5-Pro.

OCR: DocClaw achieved a best overall score of 96.45 (with Gemini-2.5-Pro) on OmniDocBench v1.6, outperforming the specialized PaddleOCR-VL-1.6 (96.34) and general-purpose VLMs. Notable improvements were seen in Formula CDM and Table TEDS metrics, attributed to visual refinement operations.
DocQA: On MMLongBench-Doc, DocClaw reached a best score of 61.5, surpassing the Gemini-2.5-Pro baseline by 3.4 points and outperforming specialized pipelines like SimpleDoc and DocDancer.
KIE: DocClaw achieved an overall 'ALL' average score of 86.9 on the combined OCRBench and OCRBench v2 subsets. Specifically, on the OCRBench v2 (EN) subset, it achieved a score of 87.5, outperforming specialized methods like Qianfan-OCR.

Ablation Studies:

State and Skills: Combining both Document State and Skills yielded the highest performance. The Document State significantly reduced latency (from ~95s to ~60s) across consecutive queries by enabling knowledge reuse, while Skills improved accuracy by providing task-oriented guidance.
Refinement: Visual refinement operations (zoom, crop, rotate) were triggered on 433 pages, with "zoom" contributing the most to performance gains, particularly for formula recognition.
Verification: In KIE tasks, incorporating OCR and verification operations corrected 35.3% and 47.7% of extraction errors made by vanilla VLMs on English and Chinese subsets, respectively.

Significance and Claims

The paper posits that DocClaw demonstrates the effectiveness of a unified agentic framework for handling heterogeneous document processing tasks. By decoupling the interaction loop and tool space from specific task definitions, DocClaw allows for flexible adaptation across OCR, DocQA, and KIE. The authors claim that the structured document state is crucial for overcoming the "knowledge silo" problem of existing methods, enabling the system to accumulate and refine information progressively. Furthermore, the use of document skills provides a mechanism to steer the agent toward appropriate strategies without requiring separate model training for each task. The results suggest that this approach can achieve competitive, and often superior, performance compared to both general-purpose VLMs and specialized, task-specific pipelines.