💬 NLP

MARDoc: A Memory-Aware Refinement Agent Framework for Multimodal Long Document QA

MARDoc is a Memory-Aware Refinement Agent framework that improves multimodal long-document QA by decoupling the process into specialized Explorer, Refiner, and Reflector agents that utilize a dynamically updated structured memory to reduce context noise and enhance multi-hop reasoning.

Original authors: Kaifeng Chen, Hongtao Liu, Qiyao Peng, Jian Yang, Yongqiang Liu, Xiaochen Zhang, Qing Yang

Published 2026-06-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kaifeng Chen, Hongtao Liu, Qiyao Peng, Jian Yang, Yongqiang Liu, Xiaochen Zhang, Qing Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a massive, 100-page mystery novel where the clues are scattered across text, charts, photos, and tables. You have a team of detectives (an AI) trying to find the answer to a specific question.

The Old Way: The "Monolithic" Detective
In most current systems, the detective works like a person who never throws anything away. Every time they look at a page, they write down the page number, what they saw, and their thoughts in a single, giant notebook.

The Problem: After 20 turns of searching, this notebook is huge. It's filled with the actual clues, but also with dead ends, irrelevant sentences, and "noise." The detective gets overwhelmed. The important clues get buried under pages of junk, making it hard to connect the dots. This is called the "context dilution" problem.

The New Way: MARDoc (The "Memory-Aware" Team)
The paper introduces MARDoc, a smarter system that doesn't just keep a giant notebook. Instead, it uses a team of three specialized detectives who work together using a dynamic, organized filing system (Structured Memory) instead of a messy notebook.

Here is how the team works, using a creative analogy:

1. The Explorer (The Scout)

Role: This detective runs around the document looking for clues. They don't just read; they use special tools to jump to specific pages, zoom in on charts, or search for keywords.
Action: They gather raw information and bring it back to the team.
Analogy: Think of them as a scout running through a forest, picking up leaves, rocks, and branches and dumping them in a pile.

2. The Refiner (The Librarian)

Role: This is the most important new part. While the Scout brings back a messy pile of "raw" evidence, the Librarian immediately sorts it.
Action: They throw away the noise (irrelevant leaves) and organize the good stuff into two neat categories:
1. Evidence Memory: The hard facts (e.g., "The chart on page 27 says 26%").
2. Reasoning Memory: The logical steps connecting the facts (e.g., "Because the chart says X, and the text says Y, we can calculate Z").
Analogy: Instead of keeping the whole pile of forest debris, the Librarian creates a clean, concise summary card. They throw away the old pile and replace it with this new, perfect card. This keeps the team's "brain" small and focused, preventing them from getting lost in the past.

3. The Reflector (The Supervisor)

Role: This detective looks at the Librarian's summary card and asks, "Do we have enough to solve the mystery?"
Action:
- If Yes: They write the final answer.
- If No: They don't just say "try again." They give specific instructions to the Scout, like, "You missed the total number of people in the survey; go find that specific table."
Analogy: The Supervisor checks the map. If the map is missing a key landmark, they tell the Scout exactly where to go next, rather than letting the Scout wander aimlessly.

Why This Matters

The paper tested this system on two difficult benchmarks (MMLongBench-Doc and DocBench) involving long, complex documents with pictures and text.

The Result: MARDoc performed significantly better than previous methods. Even when using a smaller, less powerful AI model, it beat systems using much larger models.
The Reason: By constantly cleaning up the "memory" and keeping only the essential facts and logic, the system avoids getting confused by its own history. It stays sharp and focused, just like a detective who keeps a clean case file rather than a messy stack of papers.

In short: MARDoc solves the problem of "forgetting the important stuff because there's too much junk" by having a dedicated team member whose only job is to clean, organize, and summarize the evidence after every step, ensuring the team never loses its way in a long document.

Technical Summary: MARDoc

Problem Statement

Multimodal long-document question answering (QA) is critical for applications like financial analysis and scientific literature review but remains challenging due to two primary factors: the joint expression of information across text, layout, tables, and figures, and the sparse distribution of evidence across many pages.

Existing iterative retrieval-reasoning agents face significant limitations:

Monolithic Context Stream: Most systems maintain a single, ever-growing context that mixes retrieval traces, observations, and intermediate reasoning. As interactions accumulate, key evidence becomes scattered and diluted by irrelevant history, compressing the model's effective reasoning space and introducing noise.
Lack of Memory-Grounded Quality Control: Current agentic methods often fail to distinguish between missing evidence, misinterpreted visual content, and flawed cross-page reasoning. Errors introduced in early turns persist and compound across iterations, leading to hallucinations in multi-hop reasoning chains.

Methodology: MARDoc Framework

The authors propose MARDoc (Memory-Aware Refinement Agent), a framework that replaces the monolithic context stream with a dynamically updated structured memory driven by an "Explore–Refine–Reflect" loop. The framework decouples the QA process into three specialized agents:

1. Document Processing & Outline Construction

Before agent interaction, the system employs MinerU2.5 for precise document parsing to generate fine-grained semantic nodes preserving layout structure. Qwen3-VL is used to generate concise descriptions for visual elements (images, tables), creating a unified structured outline that serves as the index for downstream retrieval tools.

2. The Explorer Agent

The Explorer iteratively retrieves multi-granularity evidence from the structured outline using a customized toolset (textual, visual, and metadata search). It operates on a ReAct framework, generating reasoning thoughts and tool actions to gather preliminary information. Unlike traditional agents that append all history to a context window, the Explorer receives only the current query, the document outline, the previous structured memory, and feedback from the Reflector.

3. The Refiner Agent

The Refiner simulates human expert reasoning by distilling interaction traces into a compact, structured memory, replacing the traditional accumulation of raw context. It processes the user query, previous memory, and the Explorer's current reasoning chain to generate two distinct components:

Evidence Memory ( $M_E$ ): A high-level log of central, query-relevant factual information (e.g., key statements, sources, condensed multimodal data) with redundant information removed.
Reasoning Memory ( $M_R$ ): A clear reasoning path integrating evidence nodes, annotated with dependencies. This decouples evidence from reasoning, allowing the system to trace whether errors stem from incorrect facts or flawed logic.

The Refiner overwrites the old memory with the updated version at each iteration, preventing context explosion.

4. The Reflector Agent

The Reflector examines the compacted structured memory to determine if the evidence is sufficient to answer the question.

If Sufficient: It outputs the final answer.
If Insufficient: It generates targeted feedback (e.g., "the total number of X is missing") to guide the Explorer toward specific missing information, preventing the repetition of past mistakes.

Key Contributions

Decoupled Retrieval and Refinement: MARDoc replaces the monolithic context stream with a dynamically updated structured memory. By decoupling information retrieval from evidence digestion, it keeps the reasoning context compact, fundamentally alleviating context dilution.
Dual-Mechanism Memory: The framework introduces a distillation process that compacts traces into Evidence Memory (factual grounding) and Reasoning Memory (logical coherence), alongside an iterative reflective assessment that verifies sufficiency and provides heuristic feedback.
Training-Free Performance: The system achieves strong results without task-specific fine-tuning, relying solely on prompt engineering and the structured memory mechanism.

Experimental Results

The authors evaluated MARDoc on two challenging benchmarks: MMLongBench-Doc and DocBench.

Performance: Using Qwen3-VL-30B as the backbone, MARDoc achieved 57.1% accuracy on MMLongBench-Doc. This outperforms same-backbone baselines (e.g., DocAgent with Qwen3-30B at 48.3%) and matches the performance of DocAgent using the much larger Claude 3.5 Sonnet (57.3%).
Robustness: MARDoc demonstrated a smaller performance decline as the number of evidence pages increased compared to other agent-based methods, indicating superior ability to integrate scattered information.
Ablation Studies:
- Removing the Refiner caused drastic performance degradation, confirming that the Reflector cannot effectively identify errors in noisy, linearly accumulated contexts without structured memory.
- Removing Reasoning Memory ( $M_R$ ) primarily hurt multi-hop question performance, while removing Evidence Memory ( $M_E$ ) hurt performance on unanswerable questions, highlighting their complementary roles.
- Replacing structured memory with plain-text compression resulted in the worst performance, validating the necessity of structured representation.
Efficiency: While MARDoc incurs a modest increase in token usage (~~22.5k additional tokens) and latency (~~16s) compared to single-pass methods, this trade-off yields a significant accuracy gain (7.8 percentage points).

Significance and Claims

The paper claims that MARDoc validates the effectiveness of structured memory for agentic document QA. By preventing the full interaction history from being repeatedly propagated, the framework maintains a compact reasoning state while preserving answer-critical facts and their logical dependencies.

The authors position MARDoc as a solution to the "context explosion" problem in iterative retrieval-reasoning, demonstrating that a memory-mediated interface can significantly enhance multi-hop reasoning capabilities in multimodal long-document tasks without requiring expensive model fine-tuning. The work suggests that the quality of the memory representation (structured vs. monolithic) is a more critical factor for long-context reasoning than simply increasing the model's context window or parameter count.