💬 NLP

CMDR: Contextual Multimodal Document Retrieval

This paper introduces CMDR, a new multimodal document retrieval task and benchmark (CMDR-Bench) that requires modeling document context, along with the CMDR-Embed framework and CMCL learning objective that jointly encode multiple pages to significantly outperform existing non-contextual methods.

Original authors: Ryota Tanaka, Taku Hasegawa, Kyosuke Nishida

Published 2026-07-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ryota Tanaka, Taku Hasegawa, Kyosuke Nishida

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific instruction in a massive, 200-page manual for a new piece of complex machinery.

The Old Way (Non-Contextual Retrieval):
Imagine you have a librarian who looks at every single page of that manual as if it were a completely separate, unrelated piece of paper. If you ask, "Where do I plug in the power cord?" the librarian might find a page that says "Power Cord" in big letters. But what if the actual answer is on page 50, which just says "See Figure 3," and Figure 3 is actually on page 12? The old librarian, looking at pages in isolation, would miss the connection. They treat the document like a stack of loose-leaf notes rather than a cohesive story.

The Problem:
Current computer systems for finding information in documents work like that old librarian. They are great at matching keywords (like "power" or "cord") but terrible at understanding the "story" of the document. They can't connect the dots between page 12 and page 50. This is a big problem because real-world documents (like manuals, research papers, and books) often spread important information across many pages.

The New Solution (CMDR):
The authors of this paper introduced a new way of thinking called CMDR (Contextual Multimodal Document Retrieval). Think of this as giving the librarian a "superpower": the ability to read several pages at once and understand how they relate to each other.

Here is how their new system, CMDR-Embed, works, using a few simple metaphors:

1. The "Sliding Window" (Reading in Chunks)

Instead of reading one page at a time, the new system reads a "window" of pages together (for example, pages 1 through 4). It's like reading a chapter of a book rather than just a single sentence.

The Trick: It slides this window down the document (reading 2–5, then 3–6, etc.). This allows the system to see the "context"—the information on the previous page that explains what's happening on the current page.

2. The "Group Study" vs. "Solo Study" (Joint Encoding)

Old Method: Every page studies alone. It memorizes its own content but doesn't know what the neighbors are talking about.
New Method: The pages in the "window" study together as a group. They share notes. This helps the system understand that "Figure 3" on page 12 refers to a diagram on page 11.

3. The "Strict Teacher" (CMCL Training)

There is a risk in this "group study" approach: if the pages talk too much to each other, they might all start sounding the same, making it hard to tell them apart later.
To fix this, the authors created a special training method called CMCL (Contextual Multimodal Contrastive Learning).

The Metaphor: Imagine a teacher giving a test. The teacher says, "You need to understand the story (context), but you also need to know exactly which page you are on."
The teacher uses "hard negatives"—pages that look very similar or are right next to each other—to force the system to learn the subtle differences. It ensures the system knows that while Page 12 and Page 13 are related, they are not the same page.

4. The New Benchmark (CMDR-Bench)

To prove this works, the authors built a new test called CMDR-Bench.

The Test: They created 800 tricky questions based on long documents (averaging 183 pages long).
The Challenge: The questions are designed so you cannot answer them by looking at just one page. You have to use clues from other pages. For example, "What is the function of the button next to the LED on the right?" requires you to look at a diagram on one page and a text description on another.
The Result: The new system (CMDR-Embed) significantly outperformed all the old "solo-study" systems. It was much better at finding the right page when the answer required connecting information across the document.

Summary

In short, the paper argues that to find information in complex, visual documents (like manuals with charts and diagrams), computers need to stop treating pages as isolated islands. Instead, they need to learn to read the "neighborhood" of pages together. By doing this, the new system can solve puzzles that require looking at the whole picture, not just a single snapshot.

What the paper does NOT claim:

It does not claim this system can replace human doctors or lawyers.
It does not claim it works perfectly on every type of document (it noted struggles with very dense research papers).
It does not claim it is a "magic bullet" for all AI; it specifically addresses the problem of retrieving pages from long, multi-page documents.

Technical Summary: CMDR: Contextual Multimodal Document Retrieval

1. Problem Definition

Existing multimodal document retrieval (MDR) benchmarks and methods primarily focus on direct retrieval, where a query is matched against individual pages using simple lexical or semantic similarity. These approaches typically encode pages independently, implicitly assuming that contextual information yields negligible benefits. However, real-world documents often distribute information across multiple pages, requiring models to resolve queries that depend on cross-page dependencies and global document structure.

Current limitations include:

Lack of Contextual Evaluation: Existing benchmarks (e.g., MMLB-Doc, ViDoRe) do not assess the ability to retrieve pages that are not explicitly mentioned in the query but are necessary to resolve it via document context.
Independent Encoding: Most state-of-the-art retrievers (e.g., ColPali, ColQwen) encode pages in isolation, failing to capture relationships between pages.
Ambiguity Resolution: Queries often require reasoning over text completion, coreference resolution, structured understanding, or multi-hop reasoning that spans multiple pages, which independent encoders cannot handle.

To address this, the authors introduce CMDR (Contextual Multimodal Document Retrieval), a task requiring systems to retrieve relevant pages from a multi-page document by leveraging the surrounding document context.

2. Methodology

A. CMDR-Bench (The Benchmark)
The authors constructed a new benchmark comprising 800 high-quality, human-annotated queries across 255 documents (average length: 183.5 pages) spanning six domains (manuals, research papers, theses, etc.). The queries are categorized into four types requiring contextual reasoning:

Text Completion (TC): Reconstructing semantic flow split across pages.
Coreference Resolution (CR): Resolving pronouns or abbreviations using cues from other pages.
Structured Understanding (SU): Understanding relationships between structured components (e.g., table headers and rows) across page boundaries.
Multi-hop Reasoning (MR): Connecting entities in the query to relevant pages via a path of contextual information.

B. CMDR-Embed (The Model)
CMDR-Embed is a contextual multimodal embedding framework designed to jointly encode multiple pages while preserving page-level discriminability.

Contextualized Multimodal Encoding: Instead of encoding pages independently, the model adopts a chunk-then-split strategy. Consecutive pages are grouped into overlapping chunks (using a sliding window with stride $s$ and window size $w$ ) and fed into a Large Vision-Language Model (LVLM). The resulting token-level representations are then split back into page-level embeddings. This allows the model to incorporate cross-page context into each page's representation.
Multimodal Late Interaction (LI): Similarity between a query and a page is computed using Late Interaction, which captures fine-grained token-level matching between query vectors and contextualized page vectors.

C. CMCL (Contextual Multimodal Contrastive Learning)
A critical challenge in joint encoding is information leakage, where pages within the same document become too similar, reducing discriminability. To solve this, the authors propose CMCL, a contrastive learning objective that balances contextual modeling with page-level distinctiveness.

Hard Negatives: The loss function incorporates two types of context-aware hard negatives:
- In-Chunk Negatives: Different pages within the same encoding chunk.
- In-Document Negatives: Other chunks from the same document.
Loss Function: The total loss is a weighted sum of a context-aware loss ( $L_{Context}$ ) and a standard in-batch contrastive loss ( $L_{Batch}$ ):
$L_{CMCL} = \lambda L_{Context} + (1 - \lambda) L_{Batch}$
This ensures the model learns to distinguish semantically similar pages within the same document while still utilizing their shared context.

3. Key Results

Experiments were conducted on CMDR-Bench using nDCG@5 as the primary metric.

Performance Gain: CMDR-Embed significantly outperforms non-contextual baselines. Using the ColQwen backbone, CMDR-EmbedQwen achieved an overall score of 56.6, representing a 16.2-point improvement over the best non-contextual baseline (ColQwen finetuned on the same data, which scored 40.4).
Ablation Studies:
- Removing the CMCL loss (using only in-batch negatives) caused a significant drop (7.2 points for CMDR-EmbedQwen), confirming the necessity of hard negatives for maintaining discriminability.
- Removing Late Interaction (replacing with mean pooling) resulted in a massive performance drop (18.5 points), highlighting the importance of fine-grained token matching.
- Both In-Chunk and In-Document negatives contributed to performance, with In-Chunk negatives being more effective for short-range context and In-Document negatives for long-range context.
Efficiency: The model achieves these gains with minimal computational overhead. Hierarchical token pooling can reduce storage by 80% and accelerate search by 3.57× with negligible performance loss.
Generalization: When trained with multitask learning (combining CMDR-Synth and ViDoRe), the model improved non-contextual retrieval performance by 4.7% while maintaining strong contextual retrieval capabilities.

4. Significance and Claims

The paper claims that context-aware multimodal embeddings are essential for advancing document retrieval in real-world scenarios.

Beyond Isolated Pages: The work demonstrates that treating documents as isolated pages is insufficient for complex queries. Explicitly modeling document context allows systems to resolve ambiguities and retrieve information that is not lexically present in the target page.
New Standard: CMDR-Bench establishes the first benchmark that jointly requires multi-page reasoning and multimodal retrieval, filling a gap left by existing datasets that focus on direct matching.
Practical Viability: The proposed method (CMDR-Embed) proves that contextual modeling can be integrated into existing LVLM-based retrieval architectures without sacrificing page-level discriminability or efficiency.

The authors conclude that future multimodal document retrieval systems must move beyond isolated page representations to explicitly model document-level context to handle the complexity of long-form, multi-page documents.