💬 NLP

MARA: A Multimodal Adaptive Retrieval-Augmented Framework for Document Question Answering

The paper proposes MARA, a multimodal adaptive retrieval-augmented framework that enhances document question answering by introducing query-aligned region encoding for precise retrieval and a self-reflective evidence controller for adaptive generation, outperforming existing state-of-the-art methods across six benchmarks.

Original authors: Hui Wu, Haoquan Zhai, Yuchen Li, Hengyi Cai, Peirong Zhang, Yidan Zhang, Lei Wang, Chunle Wang, Yingyan Hou, Shuaiqiang Wang, Dawei Yin

Published 2026-04-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hui Wu, Haoquan Zhai, Yuchen Li, Hengyi Cai, Peirong Zhang, Yidan Zhang, Lei Wang, Chunle Wang, Yingyan Hou, Shuaiqiang Wang, Dawei Yin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of a single notebook, you have a massive library filled with thousands of books, magazines, charts, and photos. Some pages are just text, others are complex diagrams, and some are a mix of both.

Your job is to answer a specific question, like: "What percentage of US employers planned to hire in the first quarter of 2018?"

The Old Way: The "Blind" Librarian

In the past, if you asked a computer to find the answer, it acted like a blind librarian.

The Search: The librarian would grab a stack of books based on a generic keyword match. They didn't really "read" the question deeply; they just grabbed the top 3 or 10 books that looked somewhat related.
The Problem:
- Too Broad: Sometimes they grabbed the whole book, missing the tiny chart on page 42 that actually had the answer.
- Too Narrow: Sometimes they grabbed a book that had the right words but the wrong context.
- The "Top-K" Trap: The librarian would force you to read exactly 3 books (or 10), no matter what. If the answer was in the 4th book, you'd miss it. If the first 3 books were full of noise, you'd get confused.

This is what the paper calls "Query-Agnostic" (ignoring the specific question) and "Static Selection" (rigidly sticking to a fixed number of books).

The New Way: MARA (The "Smart Detective")

The authors of this paper created a new system called MARA (Multimodal Adaptive Retrieval-Augmented). Think of MARA as a super-smart detective with two special tools that make them much better at solving the case.

Tool 1: The "X-Ray Goggles" (Query-Aligned Region Encoder)

When the old librarian looked at a document, they saw it as one big, blurry blob. MARA's detective puts on X-Ray goggles that can zoom in and out instantly.

How it works: When you ask a question, the detective doesn't just look at the whole page. They ask: "Does this question need the big picture (the whole chart), the middle section (a specific paragraph), or the tiny details (a single number in a table)?"
The Magic: The system learns to highlight the most important parts of the document based on your specific question. If you ask about a specific number, it zooms in on the table cell. If you ask about the general trend, it looks at the whole graph. It ignores the irrelevant noise.

Tool 2: The "Self-Checking Assistant" (Self-Reflective Evidence Controller)

This is the most clever part. In the old system, the detective would read the first 3 books and immediately try to write the report, even if they weren't sure they had enough info.

MARA's detective has a self-checking assistant who whispers in their ear while they read:

"Hey, you've read the first book. Do you have enough info to answer the question?"
If Yes: "Great! Write the answer now." (Stop early, save time).
If "Maybe": "You have some clues, but you're missing a key piece. Let's grab the next book and keep reading."
If No: "You're totally lost. Keep digging deeper."

Instead of blindly reading a fixed number of books, this assistant dynamically decides when to stop. This prevents the detective from getting overwhelmed by too much useless information (noise) or missing the answer because they stopped too early.

Why This Matters (The Results)

The researchers tested this new detective on six different types of "crime scenes" (datasets), ranging from scientific papers to slide decks and charts.

Better Accuracy: MARA found the right answers much more often than the old methods because it knew exactly where to look and when to stop.
Efficiency: It didn't waste time reading 10 books when the answer was in the first one. It saved a lot of computing power (time and money) by being smart about how much it read.
Flexibility: It works great whether the answer is in a tiny table cell or a giant infographic.

The Big Picture

In simple terms, MARA changes how computers read documents. Instead of being a rigid robot that grabs a fixed number of pages, it acts like a human expert who:

Scans a document to find the specific spot relevant to the question.
Reads just enough to be sure of the answer, stopping exactly when the job is done.

This makes AI much better at handling the messy, complex, real-world documents we use every day, from financial reports to medical charts.

1. Problem Statement

The paper addresses the limitations of current Retrieval-Augmented Generation (RAG) systems when applied to multimodal documents (e.g., technical reports, academic papers, slides containing text, tables, charts, and layouts). Existing approaches suffer from two primary bottlenecks:

Query-Agnostic Encoding: Traditional document encoders generate static representations independent of the user's query. In multimodal documents, this leads to "diluted attention," where the system fails to distinguish between coarse-grained layout structures and fine-grained visual cues (e.g., specific table cells or image regions) that are actually relevant to the question.
Static Evidence Selection: Current pipelines use a fixed top- $k$ strategy to select documents for the generator. This rigid approach fails to account for the uncertain distribution of relevant information.
- If $k$ is too small, critical evidence may be missed (partial answers).
- If $k$ is too large, the generator is overloaded with irrelevant noise, degrading answer quality.
- The system cannot dynamically adjust the amount of context based on whether the current evidence is sufficient to answer the query.

2. Methodology: The MARA Framework

The authors propose MARA (Multimodal Adaptive Retrieval-Augmented), a framework that introduces query-adaptive mechanisms to both the retrieval and generation stages. It consists of two core components:

A. Query-Aligned Region Encoder (QRE)

Goal: To improve retrieval precision by creating dynamic, query-aware document representations.

Hierarchical Decomposition: Instead of treating a document as a single block, QRE decomposes it into three granularities:
1. Global: The entire document.
2. Coarse: High-level regions (e.g., layout blocks, figure sections).
3. Fine: Localized patches (e.g., table cells, text spans, image crops).
Offline Encoding: These regions are encoded offline using a shared multimodal encoder to create embeddings ( $E_g, E_c, E_f$ ).
Online Query Alignment: During inference, the system computes attention weights between the query and the coarse/fine region embeddings.
Gated Fusion: A lightweight module predicts fusion weights to dynamically combine the global, coarse, and fine representations based on query relevance. This allows the retriever to focus on the appropriate level of granularity (e.g., focusing on a specific chart for a data question vs. a section header for a conceptual question).

B. Self-Reflective Evidence Controller (SEC)

Goal: To replace static top- $k$ selection with a dynamic, feedback-driven evidence integration strategy.

Sliding Window Mechanism: The generator processes retrieved documents using a fixed-size sliding window.
Self-Reflection Loop: At each step, the generator is prompted to evaluate the current context (current window + accumulated memory) and output a discrete signal:
- Suf. (Sufficient): The evidence is enough; generate the final answer and stop.
- Par. Suf. (Partially Sufficient): Extract useful content from the current window, append it to the memory, and continue to the next iteration.
- Insuf. (Insufficient): The current window adds no value; advance the window without updating memory.
Early Stopping: The process terminates as soon as "Suf." is detected, preventing unnecessary context overload. If all candidates are exhausted without sufficiency, the model falls back to generating from accumulated memory or abstains.

3. Key Contributions

Novel Framework: Introduction of MARA, the first framework to jointly address query-agnostic encoding and static evidence selection in multimodal RAG.
Query-Aligned Region Encoding: A multi-granular representation module that dynamically reweights document regions based on query relevance, enabling fine-grained retrieval in complex layouts.
Self-Reflective Evidence Control: A generation-time module that adaptively expands the evidence scope using a sliding-window strategy and self-monitoring, replacing rigid top- $k$ inputs with a flexible, context-aware approach.
Comprehensive Evaluation: Extensive experiments on six diverse benchmarks (ArXivQA, ChartQA, DocVQA, InfoVQA, PlotQA, SlideVQA) demonstrating consistent improvements over State-of-the-Art (SOTA) methods.

4. Experimental Results

The paper evaluates MARA against baselines like VisRAG and Oracle (using ground-truth documents) across six datasets.

Generation Performance:
- MARA consistently outperforms VisRAG. On average, it achieves 88.1% of Oracle performance with the MiniCPM-V 2.6 backbone and 87.8% with GPT-4o.
- It significantly narrows the gap to Oracle performance, particularly in tasks requiring complex reasoning (e.g., PlotQA).
- Unlike static top- $k$ methods, MARA avoids the performance drop seen when increasing $k$ (e.g., from top-3 to top-10), proving that dynamic selection filters noise effectively.
Retrieval Performance:
- The QRE module achieves state-of-the-art results in MRR@10 and Recall@10, outperforming strong VLM-based retrievers (SigLIP, ColPali, DSE) by significant margins (e.g., +3.6 MRR points on average).
Efficiency:
- MARA achieves higher accuracy than top-10 baselines while significantly reducing runtime.
- The SEC module requires an average of <2 iterations per query, demonstrating that it rarely needs to expand context extensively, thus balancing efficiency and accuracy.
Ablation Studies:
- Removing either QRE or SEC leads to notable performance drops, confirming that both retrieval adaptivity and generation adaptivity are necessary.
- The synergy between the two modules is crucial; removing both reduces performance to standard RAG baselines.

5. Significance

Paradigm Shift: MARA moves multimodal RAG from a static, "retrieve-then-generate" pipeline to a dynamic, iterative reasoning process. It mimics human behavior by selectively attending to relevant information and seeking more evidence only when necessary.
Scalability: By decoupling offline encoding from online query interaction and using early stopping, MARA offers a scalable solution for real-world applications where document complexity varies wildly.
Interpretability: The framework provides insights into why certain regions are retrieved (via attention weights) and when the model decides to stop searching (via the SEC signal), enhancing the transparency of multimodal QA systems.

In summary, MARA establishes a new standard for multimodal document QA by proving that adaptive retrieval and adaptive generation are essential for handling the complexity and heterogeneity of real-world visual documents.