Technical Summary: SaMer – Object-Evidence Preserving Token Merging for Vision-Language Retrieval

Problem Statement

Multi-vector vision-language retrieval systems, such as ColPali and ColQwen2, achieve fine-grained retrieval by preserving token-level visual evidence through maximum-similarity (MaxSim) late interaction. However, this approach incurs significant storage and computational costs because it requires storing and comparing hundreds to thousands of image-side patch embeddings for every image.

Existing token compression strategies, such as pruning or feature-based merging, attempt to reduce these costs. However, the authors identify a critical failure mode in these methods: object-instance collapse. When compression relies solely on feature similarity, visually similar patches from different object instances (e.g., two different people or cars) may be merged into a single representation. This collapses distinct evidence that future query tokens might need to select. Furthermore, pruning discards visual evidence before the query is known, potentially removing the specific tokens required for a match. The core challenge is to compress image-side tokens without losing the object- and region-level evidence necessary for query-selectable matching.

Methodology: SaMer

The authors propose SaMer (Semantic-aware Merging), an object-aware token merging framework designed to compress post-projector visual tokens into $K$ representative centroids while preserving the original late-interaction interface.

1. Feature-Spatial Soft Assignment

SaMer compresses $N$ post-projector visual tokens into $K$ merged tokens ($K \ll N$). It employs a feature-spatial soft assignment mechanism where each visual token $v_i$ is assigned to a representative $k$ based on a distance metric combining:

• Feature Similarity: $1 - v_i^\top \mu_k$, grouping semantically similar tokens.
• Spatial Coherence: $\gamma \|p_i - s_k\|_2^2$, preventing visually similar but spatially distant regions from merging aggressively.

The representatives are formed as normalized weighted centroids ($r_k$) rather than discarded tokens, ensuring the compressed representation remains compatible with MaxSim scoring.

2. Object-Aware Merge Prior (Training Only)

To prevent cross-instance mixing, SaMer utilizes object annotations only during training as a merge prior, not as an auxiliary grounding loss.

• Mechanism: The system estimates the bounding box (bbox) label distribution for each representative using a stop-gradient hard assignment.
• Penalty: An instance inconsistency penalty ($P_{inst}$) is added to the distance metric. If assigning a token to a representative would mix evidence from different object instances (i.e., the token's bbox label differs from the dominant label of the representative), the assignment weight is reduced.
• Inference: At inference time, SaMer requires no ground-truth bounding boxes or object detectors. It performs annotation-free feature-spatial soft assignment, relying on the projection layer adapted during training to implicitly preserve object structure.

3. Compression-Aware Projection-Only Adaptation

To align the compressed representation with the retrieval space, SaMer adapts only the shared projection layer while keeping the vision encoder and language backbone frozen.

• The projection layer is trained using a multi-positive InfoNCE retrieval loss over the compressed MaxSim scores ($S_K$).
• This ensures the merged tokens remain effective for late-interaction scoring without altering the backbone representations or requiring a new retrieval architecture.

Key Contributions

1. Problem Framing: The authors reframe visual-token compression for late-interaction retrieval as an evidence-preservation problem, identifying object-instance collapse as a primary failure mode of feature-based merging.
2. SaMer Framework: They propose an object-aware token merging method that uses training-time object annotations as a merge prior to discourage cross-instance merging, requiring no annotations or detectors at inference.
3. Efficiency and Performance: They demonstrate that SaMer, with $K=64$, removes over 93% of image-side tokens and reduces ColPali storage by 16.09×, while simultaneously improving retrieval performance (R@1) and phrase-level grounding compared to pruning and pooling baselines.

Experimental Results

The authors evaluated SaMer on Flickr30K, MSCOCO, ImageCoDe, and DocVQA.

• Retrieval Performance:
  • On Flickr30K, SaMer improved ColPali's R@1 from 77.0 to 82.4 and ColQwen2's R@1 from 73.6 to 79.3.
  • On MSCOCO, ColPali's R@1 improved from 47.4 to 51.6.
  • SaMer consistently outperformed compression baselines (H-Pool, HPC, SAP) under matched token budgets ($K=64$) and adaptation settings.
• Grounding Quality:
  • SaMer demonstrated superior phrase-level grounding. Metrics such as BoxMass (relevance concentration inside annotated regions) increased from 47.8 to 54.2, and CoverageIoU improved from 13.1 to 16.4.
  • This indicates that SaMer preserves query-selectable object evidence better than pruning or simple pooling, which often spread relevance to background or nearby objects.
• Efficiency:
  • For ColPali, SaMer reduced storage by 16.09× and MaxSim operations by 16.1×.
  • Query throughput (QPS) increased by 4.3× on Flickr30K and 9.1× on MSCOCO.
  • For ColQwen2, storage was reduced by 3.73× (Flickr30K) and 1.99× (MSCOCO), with corresponding throughput gains.

Significance and Claims

The paper argues that the effectiveness of retrieval compression depends not merely on reducing the token count, but on preserving the evidence future query tokens need to select.

The authors claim that:

1. Object-aware merging is superior to feature-only pooling: Simply merging similar patches can collapse distinct object instances, whereas SaMer's object-aware prior maintains the distinction necessary for fine-grained retrieval.
2. Training-time priors enable inference-time efficiency: By using object annotations only as a merge prior during training, SaMer achieves object-consistent compression without the computational overhead of running detectors during inference.
3. Projection-only adaptation is sufficient: Aligning the retrieval space via the projection layer alone allows the system to maintain high performance with frozen backbones, making the method plug-and-play for existing multi-vector retrievers.

The results suggest that efficient multi-vector retrieval requires a balance between compression and the preservation of query-selectable, object-level evidence.