Technical Summary: MINER - Mining Multimodal Internal Representation for Efficient Retrieval

Problem Statement

Visual document retrieval (VDR) aims to retrieve relevant pages from visually rich documents (e.g., slides, reports, posters) directly from their rendered images, bypassing fragile OCR-and-chunking pipelines. Current approaches face a fundamental trade-off between retrieval quality and efficiency:

1. Late-Interaction Retrievers (e.g., ColPali): These methods represent queries and documents as sets of token-level embeddings, enabling fine-grained matching. While they achieve high retrieval quality, they incur massive index footprints (hundreds to thousands of vectors per page) and high serving costs.
2. Dense Single-Vector Retrievers: These compress an entire document into a single final-layer embedding, offering compact indices and low latency. However, they consistently lag in quality because the compression into a single vector creates an information bottleneck, discarding retrieval-relevant signals present in intermediate network layers.

Recent studies in NLP and vision suggest that internal representations of deep networks often contain richer task-relevant information than final outputs alone. This raises the question: Can the internal representations of single-vector retrievers be leveraged to recover lost retrieval signals without sacrificing the efficiency of the single-vector paradigm?

Methodology: MINER

The authors propose MINER (Mining Multimodal Internal RepreseNtation for Efficient Retrieval), a lightweight plug-in module designed to extract and fuse retrieval-relevant signals from internal transformer layers. MINER operates without modifying the backbone model, preserving the storage and serving efficiency of dense retrieval. It consists of two main stages:

1. Layerwise Diagnostic and Analysis

Before designing the module, the authors conducted a systematic analysis of single-vector retrievers (Jina, Eager, MoCa) using two metrics:

• Normalized Centered Kernel Alignment (CKA): Measures structural similarity between a layer's representation space and the final cross-modal retrieval space. This identifies a candidate set of layers ($S_{cand}$) that are structurally close enough to the final space to be amenable to lightweight extraction.
• Alignment Ratio (AR): Defined as the ratio of sample-level cosine similarity to the CKA score. This distinguishes between layers that are merely structurally similar and those that are directly coordinate-aligned with the final space.
  • Result: The analysis revealed two distinct regimes within the candidate layers. Earlier layers exhibit low AR (structurally similar but misaligned), while the final few layers exhibit high AR (directly aligned). This finding motivates differentiated extraction strategies for different layers.

2. The MINER Architecture

MINER operates in two stages:

Stage 1: Retrieval-Aligned Layer Probing
A lightweight probe is attached to each layer in the candidate set $S_{cand}$. The probe parameterization is determined by the layer's alignment regime:

• Base Probe (for high-AR layers): Applies element-wise reweighting ($p_l \odot x$) to preserve the original coordinate system.
• Normalized Projection Probe (for low-AR layers): Applies element-wise reweighting followed by a row-normalized linear projection ($\tilde{W}_l(p_l \odot x)$) to realign the misaligned subspace to the anchor space without rescaling.
• Training: All probes are trained using a cross-modal siamese objective (InfoNCE loss) to align the probed text/vision readouts with their cross-modal final-layer anchors.

Stage 2: Adaptive Sparse Multi-Layer Fusion
The probed signals are aggregated into a single dense embedding:

• Performance-Adaptive Neuron-Level Masking: A layer-specific retention ratio ($P_l$) is calculated based on the standalone validation performance (nDCG@5) of each layer. A hard binary mask retains only the top-$P_l$ neurons (based on the learned importance vector $p_l$) for each layer, suppressing irrelevant dimensions.
• Cross-Layer Fusion: The masked signals from all selected layers are combined via a learned weighted sum with a global bias:
  $$e = \sum_{l \in S_{cand}} u_l \odot h^{(l)} + b$$
  where $h^{(l)}$ is the processed readout (either masked directly or masked after projection).

Key Contributions

1. Diagnostic Insight: The authors reveal that retrieval-relevant signals in single-vector retrievers are unevenly distributed across transformer layers and require qualitatively different extraction strategies (simple reweighting vs. projection) based on an alignment ratio diagnostic.
2. MINER Module: A plug-in architecture comprising Retrieval-Aligned Layer Probing and Adaptive Sparse Multi-Layer Fusion. It improves retrieval quality without modifying the backbone or increasing storage/serving costs.
3. Empirical Performance: MINER consistently outperforms existing dense single-vector retrievers across multiple backbones and the ViDoRe benchmark suite.

Experimental Results

Experiments were conducted on the ViDoRe V1, V2, and V3 benchmarks using Jina-Embeddings-v4, Eager-Embed-v1, and MoCa-3B as backbones.

• Retrieval Quality: MINER achieved up to a 4.5% improvement in nDCG@5 over its corresponding dense backbone (e.g., MINER-Eager improved from 56.0 to 58.5 on ViDoRe V2). In several settings, it substantially narrowed the performance gap to strong late-interaction baselines (reducing the gap to 0.2 nDCG@5 in some cases).
• Efficiency:
  • Storage: MINER maintains the exact same storage footprint as the dense baseline (single-vector index), which is 42.4× smaller than late-interaction methods.
  • Latency: MINER incurs only a 1.07× query latency overhead compared to the dense baseline, while remaining 5.3× faster than late-interaction methods.
• Ablation Studies: Removing the alignment-ratio partition (using only Base or only Norm probes) or removing the sparse masking resulted in lower performance, confirming that both the differentiated probing and the adaptive masking are necessary components.
• Robustness: The method is robust to key hyperparameters, specifically the CKA cutoff for layer selection and the sparsity floor for masking.

Significance and Claims

The paper claims that MINER successfully bridges the gap between the high quality of late-interaction retrieval and the high efficiency of dense single-vector retrieval. By mining internal representations, MINER recovers information lost to the final-layer bottleneck without the storage and latency penalties associated with multi-vector approaches.

The authors position MINER as a practical solution for visual document retrieval systems that require both high accuracy and low operational costs. They note that the method is currently evaluated on models where internal hidden states and training data are accessible, serving as a lightweight enhancement to existing dense retrieval pipelines rather than a replacement for the backbone architecture itself.