💬 NLP

Unveil: Unified Visual-Textual Integration and Distillation for Multi-modal Document Retrieval

This paper proposes Unveil, a novel framework that integrates visual and textual features for robust multi-modal document retrieval and employs knowledge distillation to transfer this capability to an efficient, parsing-free visual-only model.

Original authors: Hao Sun, Yingyan Hou, Jiayan Guo, Bo Wang, Chunyu Yang, Jinsong Ni, Yan Zhang

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hao Sun, Yingyan Hou, Jiayan Guo, Bo Wang, Chunyu Yang, Jinsong Ni, Yan Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific page in a massive library of mixed media: some pages are just text, some are complex charts, some are photos with captions, and some are a chaotic mix of all three.

The Problem: The "One-Size-Fits-None" Search
Currently, search engines for these documents are stuck in a dilemma:

The "Text-Only" Librarian: This librarian is great at reading words. They can instantly find a document if you ask for a specific sentence. But if the document is a chart or a diagram with no words, or if the text is messy and hard to read, this librarian gets confused. They often miss the big picture because they ignore the visual layout.
The "Visual-Only" Librarian: This librarian looks at the whole picture. They understand charts, colors, and where things are placed on a page. But they struggle to read the fine print. If you ask for a specific word hidden in a tiny label, they might miss it because they aren't great at "reading" the text inside the image.

The Solution: "Unveil"
The researchers behind Unveil (Unified Visual-Textual Integration and Distillation) built a new system that acts like a super-librarian who can do both jobs perfectly, and then teaches a simpler assistant to do the job almost as well without needing to read.

Here is how they did it, using a simple analogy:

Step 1: The "Master Chef" (Visual-Textual Model)

First, they trained a "Master Chef" (the Teacher Model).

How it works: This chef gets two ingredients: the image of the document and the text extracted from it (using a tool called OCR, which is like a scanner that turns pictures of words into digital text).
The Result: Because the chef has both the picture and the words, they understand the document perfectly. They know what the chart looks like and exactly what the numbers say. This chef is incredibly smart but takes a long time to cook because they have to process both ingredients.

Step 2: The "Apprentice" (Visual-Only Model)

Next, they wanted a faster, cheaper assistant (the Student Model) who could work without the text ingredient.

The Challenge: If you just tell the apprentice to look at the picture, they usually miss the subtle details the Master Chef caught because they can't read the text.
The Magic Trick (Knowledge Distillation): Instead of just telling the apprentice "This is a chart," the Master Chef teaches them by showing them how they think.
- Alignment: The apprentice tries to mimic the Master Chef's "mental map" of the document.
- Soft Labels: The Master Chef doesn't just say "Yes, this is the right answer." They say, "This answer is 90% right, that one is 10% right." This helps the apprentice learn the nuance of the search.
- Adaptive Re-Weighting: If the apprentice gets a specific document wrong, the Master Chef highlights that specific mistake and says, "Pay extra attention to this one!"

The Result: Two Modes for Every Need

Once the training is done, the Unveil system offers two ways to search, depending on what you need:

The "Precision Mode" (Visual-Textual): When you need the absolute best accuracy and don't mind waiting a little longer, you use the Master Chef. They look at the image and the text to find exactly what you need.
The "Speed Mode" (Visual-Only): When you need to search thousands of documents instantly and can't wait for the text scanner (OCR) to run, you use the Apprentice. Because the Master Chef taught them so well, the Apprentice can find the right document just by looking at the image, almost as accurately as the Master Chef, but much faster.

Why This Matters

The paper shows that this new system beats the old "Text-Only" and "Visual-Only" librarians in almost every test.

It's better at finding documents with complex charts than the text-only searchers.
It's better at finding specific words in text-heavy documents than the visual-only searchers.
Most importantly, the "Speed Mode" (the Apprentice) is so good that you don't need the slow text scanner anymore for many tasks, saving time and computer power while still getting great results.

In short, Unveil creates a smart system that learns to read and see simultaneously, then teaches a faster version to do the job by just looking, bridging the gap between understanding words and understanding pictures.

Technical Summary: Unveil: Unified Visual-Textual Integration and Distillation for Multi-modal Document Retrieval

Problem Statement

Real-world document retrieval faces significant challenges due to the diversity of document formats, which include text, images, charts, and complex visual layouts. Current approaches exhibit distinct limitations depending on their modality:

Text-Based Methods: Traditional pipelines rely on Optical Character Recognition (OCR) to convert documents into machine-readable text, followed by lexical (e.g., BM25) or embedding-based retrieval (e.g., DPR). These methods introduce computational overhead and potential recognition errors. Crucially, they often discard crucial visual contextual elements (layout, graphics) essential for comprehension.
Visual-Based Methods: Recent parsing-free approaches utilize Vision-Language Models (VLMs) to process document screenshots directly. While effective at preserving structural and graphical information, empirical analysis reveals they struggle to capture fine-grained textual semantics in text-rich scenarios.

The core challenge identified is that text-based methods excel at linguistic semantics but overlook layout, while purely visual methods preserve context but lack deep language understanding. No existing method effectively adapts across both text-rich and visual-rich scenarios.

Methodology: The Unveil Framework

The authors propose Unveil (Unified Visual-Textual Integration and Distillation), a framework designed to bridge the gap between visual-textual and visual-only approaches. The framework operates in two stages:

1. Visual-Textual Embedding (Teacher Model)

The first component is a dual-encoder architecture that jointly processes document images and OCR-derived text.

Input: A document image is processed by a vision encoder to generate visual tokens. Simultaneously, OCR extracts a textual description.
Integration: These are concatenated with a prompt (e.g., <s><img> <description> What is shown in this image?</s>) and fed into a language model.
Representation: The model generates embeddings using the end-of-sequence token (</s>) from the last hidden state.
Training: The model is optimized using InfoNCE loss to maximize the similarity between relevant query-document pairs.

2. Knowledge Distillation (Teacher to Student)

To enable efficient, parsing-free retrieval, the semantic understanding of the teacher (visual-textual) is transferred to a student (visual-only) model. The student model receives only the document image as input during inference. The distillation process employs three specific strategies:

Representation Alignment: The student model is trained to minimize the distance between its query and document representations and those of the teacher model ( $L_{align}$ ). This forces the visual-only model to encode textual semantics implicitly.
Soft Label Distillation: Instead of hard one-hot labels, the teacher provides a fine-grained probability distribution over the document corpus. The student is trained to match this distribution using Kullback-Leibler (KL) divergence ( $L_{soft}$ ), allowing it to learn nuanced similarity relationships.
Adaptive Re-Weighting: The framework dynamically identifies instances where the student and teacher models disagree significantly. These instances are assigned higher weights during training to focus the student's learning on difficult cases.

The total loss function combines these components: $L_{total} = \sum w_i \times (L_{align} + L_{soft})$ .

Inference Modes

Unveil offers flexibility based on application needs:

Visual-Textual Mode: Uses both OCR and images for optimal performance in high-precision scenarios.
Visual-Only Mode: Uses the distilled visual model without OCR, reducing computational overhead while maintaining competitive accuracy.

Key Contributions

Identification of Modality Gaps: The authors demonstrate that existing text-based and visual-based methods fail to adapt effectively across different document scenarios (text-rich vs. visual-rich).
Unified Framework (Unveil): Introduction of a visual-textual embedding approach that integrates both modalities to create comprehensive document representations.
Distillation Strategies: Proposal of specific techniques (Representation Alignment, Soft Label Distillation, Adaptive Re-Weighting) to transfer robust textual understanding from a visual-textual teacher to a purely visual student, enabling parsing-free retrieval without accuracy loss.
Empirical Validation: Extensive experiments showing that the visual-textual method outperforms existing baselines, and the distilled visual-only model successfully bridges the performance gap between visual-textual and visual-only approaches.

Experimental Results

The framework was evaluated on 12 datasets covering two primary scenarios: Visual Document Retrieval (e.g., MP-DocVQA, ChartQA, SlideVQA) and Web-Page Retrieval (e.g., NQ, TriviaQA, HotpotQA).

Performance Superiority: The Unveil visual-textual model consistently achieved the highest retrieval performance (measured by Recall@10 and MRR@10) across all scenarios, outperforming both pure text-based models (like BM25, BGE-Large) and pure visual-based models (like SigLIP, ColPali, DSE).
Distillation Effectiveness: The distilled visual-only model achieved performance comparable to the teacher model and significantly outperformed existing visual-only baselines. For instance, in Visual Document Retrieval, the distilled model achieved an average Recall of 87.06%, surpassing the best visual-only baseline (VisRAG at 86.08%) and approaching the teacher's 86.93%.
Ablation Studies: Removing any component of the distillation process (Alignment, Soft Labels, or Re-Weighting) resulted in performance degradation, confirming the necessity of each strategy.
Text Length Analysis: The study noted that while increasing text length improves the teacher's performance, it incurs inference costs. The distillation process allows the student to capture the benefits of long-context understanding without the associated computational burden.

Significance and Claims

The paper claims that Unveil addresses the fundamental trade-off between semantic fidelity and computational efficiency in multi-modal document retrieval. By successfully transferring the "textual understanding" capabilities of an OCR-dependent model into a purely visual model, Unveil enables parsing-free retrieval that does not compromise on accuracy.

The authors emphasize that their approach offers a flexible system: users can choose the visual-textual mode for maximum precision in complex scenarios or the distilled visual-only mode for efficiency-critical applications. The work demonstrates that knowledge distillation is a viable path to overcoming the limitations of single-modality approaches, effectively unifying the strengths of both text and vision for robust document retrieval.