💻 computer science

VLD-RAG: Agentic Vision-Language Retrieval-Augmented Generation for Long, Visually-Rich Multi-Page Documents

VLD-RAG is an agentic multimodal retrieval-augmented generation framework that utilizes page-preserving indexing, hybrid retrieval strategies, and a coordinated multi-agent workflow to effectively answer questions by synthesizing scattered textual and visual evidence across long, visually-rich multi-page documents.

Original authors: Seonok Kim

Published 2026-07-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Seonok Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a giant, 100-page mystery, but the clues are scattered everywhere. Some clues are written in plain text, some are hidden inside complex charts, others are tucked away in diagrams, and some are just part of the page's layout. This is the world of "visually-rich documents"—think of them as the messy, colorful, and information-dense reports, manuals, and slide decks we see in the real world. For a long time, computers trying to read these documents had a major problem: they often tried to strip away all the pictures and layouts to just read the words. It was like trying to understand a comic book by only reading the dialogue bubbles and ignoring the drawings; you'd miss the context, the jokes, and the plot twists.

To fix this, scientists use a technique called Retrieval-Augmented Generation (RAG). Think of RAG as a super-smart librarian who doesn't just memorize books but goes out to find the exact pages you need before answering your question. However, when the "books" are these messy, multi-page visual documents, standard librarians often get lost. They might grab the wrong page because they only looked at the text, or they might miss a crucial chart because they didn't know how to "see" the image. The big question is: How do we build a system that can hunt down the right evidence across dozens of pages, mixing text and images perfectly, to answer a question correctly?

Enter VLD-RAG, a new framework designed to be the ultimate detective for these long, visual documents. The researchers behind this work realized that to solve a mystery spread across 150 pages, you can't just rely on one trick. Instead, they built an "agentic" system—a team of AI specialists working together. Imagine a trio of detectives: one is a Keyword Hunter who scans for exact words and numbers; another is a Visual Sleuth who looks at the overall "vibe" and layout of the pages; and a third is a Critical Verifier who checks their work.

Here's how they work together. First, the system breaks down your question into a plan. It doesn't just ask, "What is the profit?" It asks, "Find the table in Section 3, look for the chart titled 'Q4 Results,' and check the text for the specific dollar amount." Then, the Keyword Hunter and Visual Sleuth go out and search the document simultaneously. The Hunter grabs pages with the right words, while the Sleuth grabs pages that look like they contain the answer, even if the words are different.

The magic happens when they compare notes. If the Hunter says, "Page 12 looks good," but the Sleuth says, "Page 12 looks totally wrong," the system doesn't just guess. It uses a special "consistency check" to realize something is off. If the evidence feels weak or missing, the Critical Verifier steps in and says, "Hey, we missed something! Let's try asking the question a different way." This triggers a second search, refining the clues until they find the right pages. Finally, the system gathers the evidence—snippets of text, crops of charts, and page numbers—and feeds them to a generator to write the final answer, making sure every claim is backed up by the actual document.

The researchers tested this team of detectives on two massive challenges: MMLongBench-Doc and LongDocURL. These are like the "Olympics" for document reading, containing hundreds of documents with thousands of pages, complex tables, and tricky questions. The results were impressive. VLD-RAG didn't just find the right pages more often than previous methods; it found more of the right pages. On the LongDocURL benchmark, which features documents averaging 85.6 pages, VLD-RAG managed to retrieve the correct evidence pages in 81.16% of cases when looking at the top 5 results (Recall@5), beating all other methods. It also ranked the most relevant pages higher than anyone else, meaning the answer was usually sitting right at the top of the list.

The paper suggests that this success comes from not forcing the computer to choose between "reading" and "seeing." By keeping the visual layout and the text separate but working together, and by letting the AI agents double-check each other, the system avoids the mistakes that happen when you try to flatten a rich, colorful document into plain text. While the researchers note that this approach is specifically for documents where information is scattered across multiple pages, the findings strongly suggest that for these complex, multi-page mysteries, a team of specialized, verifying agents is far superior to a single, lone-wolf search.

Technical Summary: VLD-RAG

Problem Statement

Visually-rich documents (e.g., reports, slides, manuals) often distribute the evidence required to answer a question across multiple pages, mixing text with complex layouts, tables, charts, and figures. Standard Retrieval-Augmented Generation (RAG) pipelines, designed primarily for text-centric settings, degrade in these scenarios. Converting pages to plain text often results in information loss, obscuring layout semantics, spatial relationships, and non-textual cues essential for correct interpretation. Furthermore, existing vision-first approaches face challenges: reliance on extracted text or OCR can miss structural signals; naive fusion of lexical and semantic signals can be brittle when modalities disagree; and single-shot retrieval is often insufficient for long documents requiring multi-step evidence discovery scattered across pages.

Methodology: VLD-RAG Framework

The authors propose VLD-RAG, an agentic multimodal RAG framework designed for multi-page evidence retrieval and cross-page reasoning. The system operates in three primary stages:

1. Dual-Space Document Representation

VLD-RAG employs a dual-index architecture to preserve complementary signals:

Visual Index: Page images are encoded into dense vision-language embeddings ( $v_i$ ) using a vision encoder (ColPali). This preserves visual layout and textual semantics in a unified space, adhering to the "retrieve by page image" philosophy.
Text Index: Pages are parsed into structured Markdown, preserving layout information (tables, figures, headings). These are serialized and indexed as chunk embeddings ( $t_{i,j}$ ). This allows for faithful reconstruction of evidence and precise lexical matching (IDs, numbers, rare entities).

2. Multimodal Query Formulation and Hybrid Retrieval

Upon receiving a query $q$ , the system performs multimodal query decomposition to generate structured retrieval intents, including salient entities, visual anchors, and expected evidence types. The retrieval process combines two branches:

Sparse Lexical Retrieval: Extracts keywords ( $k$ ) from the query to perform standard term-frequency-based search.
Dense Semantic Retrieval: Generates a "hypothetical document" ( $h$ ) conditioned on the query using a language model (Gemma3n). This synthetic document approximates the content of an ideal answer-bearing page, bridging the semantic gap between the query and the document. The hypothetical document is encoded into a dense vision-language embedding ( $e_h$ ) and matched against page embeddings via ANN search.

Modality-Consistent Fusion:
Candidates from both branches are merged. To handle cross-modal disagreement, a consistency-constrained fusion score is applied:
$s(i) = \alpha s_{sparse}(i) + (1 - \alpha) s_{dense}(i) - \lambda \Delta(i)$
where $\Delta(i)$ penalizes candidates where sparse and dense signals significantly disagree.

3. Agentic Workflow and Iterative Refinement

VLD-RAG utilizes a verifier-guided agent workflow involving a Retrieval Agent, Answer Agent, and Validation Agent.

Evidence-Grounded Generation: The generator produces an answer grounded in retrieved page regions, parsed Markdown spans, and metadata.
Verification Loop: A validation agent checks if the generated answer is sufficiently supported by the evidence (confidence threshold $\tau$ ). If validation fails (e.g., missing citations or low confidence), the system triggers an iterative refinement step. The query is refined ( $q'$ ) based on the missing evidence, and the retrieval process repeats with updated intents.

Key Contributions

Dual-Index Architecture: A vision-language document RAG framework that preserves both visual layout cues (via page-level embeddings) and precise lexical signals (via structured Markdown serialization).
Modality-Consistent Hybrid Retrieval: A retrieval strategy that explicitly models agreement between sparse and dense modalities, using consistency constraints to improve robustness against cross-modal mismatches.
Verifier-Guided Agentic Loop: A mechanism enabling reliable multi-page evidence discovery through iterative query refinement, addressing the insufficiency of single-shot retrieval for complex, cross-page questions.

Experimental Results

The framework was evaluated on two multimodal long-document benchmarks: LongDocURL and MMLongBench-Doc.

Retrieval Performance: VLD-RAG demonstrated consistent improvements over baselines (including M3DocRAG, MDocAgent, and MoLoRAG) in Recall@K, NDCG, and MRR across all top-K settings ( $K \in \{1, 3, 5\}$ $K \in {1, 3, 5}$ ).
- On MMLongBench-Doc, VLD-RAG achieved the highest Recall and ranking quality metrics.
- On LongDocURL (which features longer documents and higher percentages of multi-page questions), VLD-RAG showed particularly strong gains in Recall and NDCG, outperforming text-only and image-only baselines.
Analysis: The results indicate that the hybrid approach effectively captures position-aware relevance, which is critical for multi-page retrieval. The integration of hypothetical document generation helped bridge semantic gaps, especially when evidence was embedded in complex layouts. While some baselines achieved higher Precision at intermediate K values, VLD-RAG's superior Recall and ranking metrics suggest better overall effectiveness for retrieval-augmented generation tasks that require comprehensive evidence coverage.

Significance

The paper posits that coordinated agent verification and multimodal hybrid retrieval are crucial for reliable grounding when correct answers depend on evidence scattered across pages. By preserving visual layout information while maintaining precise lexical matching, VLD-RAG addresses the limitations of existing RAG systems in visually-rich, long-document contexts. The findings highlight that effective multimodal RAG requires not just better encoders, but also structural strategies for fusing modalities and iterative mechanisms to recover missing evidence.