Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework

This paper introduces Prune-then-Merge, a novel two-stage framework that first adaptively filters low-information patches and then hierarchically merges the remaining embeddings to overcome the trade-off between compression efficiency and feature fidelity in multi-vector Visual Document Retrieval, achieving superior performance across 29 datasets.

The Gist

Imagine you have a massive library of visual documents—think of complex financial reports, colorful slide decks, scientific papers, and handwritten notes. You want to build a search engine that can find the right page when you ask a question like, "Show me the chart about Q3 profits."

This is the world of Visual Document Retrieval (VDR).

The Problem: The "Too Much Information" Bottleneck

In the past, computers tried to read these documents by turning them into text (like a scanner). But that misses the point: the layout, the charts, and the images are often the most important parts.

Modern AI (called Large Vision-Language Models) is great at "seeing" these documents. It breaks a single page into hundreds of tiny puzzle pieces (patches) and creates a unique "fingerprint" (embedding) for each piece. This allows the computer to match your question to the exact spot on the page where the answer lives.

But here's the catch:
If you have 1,000 documents, and each page is broken into 500 pieces, you now have 500,000 fingerprints to store and search through. It's like trying to find a specific needle in a haystack, but the haystack is made of 500,000 other needles. It's too slow and takes up too much memory.

The Old Solutions: The "Scissors" and the "Blender"

Researchers tried to fix this with two main strategies, but both had flaws:

1. The Scissors (Pruning): This method tries to cut out the "boring" parts of the page (like blank white space or decorative borders) and throw them away.
   • The Flaw: If you cut too much, you accidentally throw away the answer. It's like trying to save space in a suitcase by cutting out your socks, but then you realize you needed them for the trip. At high compression rates, the search engine gets confused and fails.
2. The Blender (Merging): This method takes groups of patches and smushes them together into one average "super-patch."
   • The Flaw: If you blend a "profit chart" with a "blank margin," the result is a muddy, useless average. You lose the sharp details needed to find the answer. It's like blending a steak and a salad; you get a smoothie that tastes like neither.

The New Solution: PRUNE-THEN-MERGE

The authors of this paper propose a clever two-step process called PRUNE-THEN-MERGE. Think of it as "Refine, then Compress."

Step 1: The Smart Filter (Pruning)

First, the system acts like a very picky editor. It looks at the document and asks, "Which parts actually matter?"

• It uses the AI's internal "attention" (like a spotlight) to identify the important patches (text, charts, figures).
• It cuts out the noise (blank spaces, logos, decorations) before doing anything else.
• Analogy: Imagine you are packing for a trip. Instead of just throwing everything in a bag, you first lay everything on the bed and remove the things you definitely won't need (like your winter coat in July). You are left with a pile of only the essential items.

Step 2: The Smart Grouping (Merging)

Now, you have a smaller pile of only the important stuff. The system then takes these high-quality items and groups similar ones together.

• It uses a technique called clustering to find patches that are talking about the same thing (e.g., all the patches describing the "Revenue" section).
• It merges these similar patches into a single, strong "summary" vector.
• Analogy: Because you already removed the junk in Step 1, you can now safely group your remaining items. You can bundle your "socks" together and your "shirts" together without worrying that you've mixed in a "toaster." The resulting bundles are clean, organized, and easy to search.

Why This is a Game-Changer

The magic of this method is the order of operations.

• If you Blend first (Merging), you mix the signal with the noise, creating a muddy mess.
• If you Cut too much (Pruning), you lose the signal entirely.
• PRUNE-THEN-MERGE does the cutting first to get a clean signal, and then blends the clean signal.

The Result:
The researchers tested this on 29 different datasets (from medical reports to legal documents). They found that:

1. They could compress the data by 50% to 80% (saving huge amounts of storage space).
2. The search engine remained almost as accurate as the original, uncompressed version.
3. Even when they pushed the compression to extreme levels (where other methods failed completely), this method kept working.

The Bottom Line

Imagine you want to send a photo to a friend over a slow internet connection.

• Old Way: You try to shrink the whole photo at once, and it comes out pixelated and blurry.
• PRUNE-THEN-MERGE Way: You first crop out the boring sky and background (Pruning), leaving only the person. Then, you compress just that person (Merging). The result is a tiny file that still looks sharp and clear.

This paper gives us a blueprint for making powerful AI search engines fast, cheap, and practical for real-world use, without sacrificing the ability to "see" and understand complex documents.

Technical Summary

1. Problem Statement

Visual Document Retrieval (VDR) aims to retrieve relevant pages from vast corpora of visually rich documents (e.g., reports, slides, academic papers) based on queries combining text and visual cues.

• The Shift: The field has moved from single-vector (page-level) models to multi-vector (patch-level) paradigms (e.g., ColPali, ColQwen). These models represent a document as a collection of patch embeddings, allowing for fine-grained "late-interaction" matching (e.g., MaxSim) that captures layout and localized details better than single-vector approaches.
• The Bottleneck: Multi-vector models suffer from prohibitive storage and computational overhead. Storing hundreds or thousands of vectors per page makes large-scale deployment impractical.
• The Trade-off Dilemma: Existing efficiency methods fall into two categories, both with significant limitations:
  1. Pruning-based: Discards low-information patches (e.g., DocPruner). While effective at moderate compression, performance suffers a sharp "cliff" at high compression ratios (e.g., >70-80%) because aggressive pruning removes critical semantic content.
  2. Merging-based: Combines multiple patches into fewer vectors (e.g., Light-ColPali). While it degrades gracefully at high compression, the naive merging process causes feature dilution, where distinctive semantic signals are averaged out, leading to unstable lossless performance ranges.

2. Methodology: PRUNE-THEN-MERGE

The authors propose PRUNE-THEN-MERGE, a novel two-stage, query-agnostic, offline compression framework that synergizes pruning and merging to overcome the limitations of using either method alone. The core logic is "First Refine, Then Compress."

Stage 1: Adaptive Pruning (Refinement)

• Goal: Filter out low-information patches (noise) such as whitespace, decorative elements, or background, leaving a high-signal set of embeddings.
• Mechanism:
  • Leverages the internal attention mechanism of the Large Vision-Language Model (LVLM).
  • Extracts attention weights from the final transformer layer, specifically focusing on the attention received by a global token (e.g., [EOS]) from each patch.
  • Computes a document-specific adaptive threshold ($\tau_d = \mu_d + k \cdot \sigma_d$) based on the mean and standard deviation of these importance scores.
  • Retains only patches with importance scores exceeding the threshold.
• Outcome: A refined, high-Signal-to-Noise Ratio (SNR) set of embeddings ($D'$).

Stage 2: Hierarchical Merging (Compression)

• Goal: Compress the refined set ($D'$) further without diluting the remaining semantic features.
• Mechanism:
  • Applies hierarchical agglomerative clustering (e.g., Ward's method) on the pre-filtered set $D'$.
  • Embeddings are normalized, and a pairwise cosine distance matrix is computed.
  • The set is partitioned into a target number of clusters ($N''_p$) based on a merging factor ($m$).
  • A new representative embedding (centroid) is generated for each cluster by computing the mean of its members.
• Outcome: A final, highly compressed set of semantically rich centroids ($D''$).

Theoretical Basis

The framework is grounded in Information Bottleneck (IB) and Rate-Distortion (RD) theory:

1. Pruning acts as a query-agnostic information filter, approximating the removal of the noise set ($D_{noi}$) to preserve the signal set ($D_{sig}$).
2. Merging acts as an optimal vector quantization step on the clean signal, minimizing distortion (MSE) by finding centroids that best represent the high-SNR data.
3. Synergy: By removing noise before merging, the resulting centroids are unbiased estimators of the true semantic concepts, avoiding the distortion caused by averaging noise with signal (a flaw in single-stage merging).

3. Key Contributions

1. Novel Framework: Introduction of PRUNE-THEN-MERGE, the first framework to explicitly combine adaptive pruning and hierarchical merging in a sequential pipeline for VDR.
2. Extended Near-Lossless Range: The method significantly extends the "near-lossless" compression range. It maintains high performance up to 60-70% compression, whereas pruning-only methods typically fail beyond 50-60%.
3. Superior High-Ratio Performance: At aggressive compression rates (80-90%), the framework avoids the performance cliffs seen in pruning methods and the feature dilution of merging methods, consistently outperforming baselines.
4. Robustness: Demonstrates effectiveness across diverse document types, languages (multilingual), and complex semantic settings (e.g., rephrased queries in RAG scenarios).

4. Experimental Results

The framework was evaluated on 29 datasets across 6 major benchmarks (ViDoRe-V1/V2, JinaVDR, REAL-MM-RAG, ViDoSeek, MMLongBench-Doc) using three state-of-the-art base models: ColQwen2.5, ColNomic, and Jina-v4.

• Performance vs. Compression:
  • Near-Lossless Range: PRUNE-THEN-MERGE extends the near-lossless range by an average of 10 percentage points compared to the best pruning baseline (DocPruner), moving from [50-60%] to [60-70%].
  • High Compression: At 80-90% compression, it consistently outperforms all baselines. For example, on ViDoRe-V1 with ColQwen2.5 at ~84% compression, it achieves an nDCG@5 of 0.86, while DocPruner drops to 0.77.
• Multilingual Generalization: The method generalizes robustly across 9 languages (including Chinese, Japanese, Hindi), maintaining superior performance-compression trade-offs where pruning-only methods often collapse.
• Complex Settings: In the REAL-MM-RAG benchmark (testing deep semantic understanding with rephrased queries), the framework excels, particularly on dense, text-heavy documents where aggressive pruning fails.
• Efficiency Gains:
  • Storage: Achieves an average storage reduction of ~54.6% (up to 58.88% for ColQwen2.5).
  • Latency: Increases offline encoding latency slightly (from ~0.46s to ~0.69s per page) due to the two-stage processing, but this remains acceptable for offline indexing. Online retrieval latency is drastically reduced due to the smaller search space.

5. Significance

This work addresses a critical barrier to the industrial adoption of multi-vector VDR models: scalability. By providing a blueprint for "refining then compressing," PRUNE-THEN-MERGE enables the deployment of high-accuracy, fine-grained retrieval systems on large-scale document corpora without the prohibitive storage costs previously associated with multi-vector approaches. It resolves the long-standing trade-off between compression rate and feature fidelity, making efficient, high-performance visual document search practically viable for enterprise and domain-specific RAG applications.