CLIP Tricks You: Training-free Token Pruning for Efficient Pixel Grounding in Large VIsion-Language Models

Technical Summary: CLIP Tricks You: Training-free Token Pruning for Efficient Pixel Grounding in Large Vision-Language Models

Problem Statement
Large Vision-Language Models (LVLMs) face significant computational overhead due to the sheer volume of visual tokens they process, which often outnumber text tokens by a large margin (e.g., 576 visual tokens vs. <100 text tokens in LLaVA). While recent token pruning methods have successfully reduced this load for general image understanding tasks, they struggle with pixel grounding tasks, such as referring expression segmentation. Existing approaches typically rely on text-agnostic visual importance (e.g., similarity to a global [CLS] token) or assume that high visual-text similarity indicates relevance. However, the authors observe that these methods fail to preserve tokens within the specific referent regions defined by the input text, leading to degraded segmentation performance.

Methodology: LiteLVLM
The paper introduces LiteLVLM, a training-free, text-guided token pruning strategy designed specifically for efficient pixel grounding. The method is built upon an in-depth analysis of the CLIP model, revealing two critical phenomena:

1. Visual-Text Similarity Reversal: Contrary to intuition, visual tokens located within the referent regions (the objects described by the text) exhibit low similarity to the text representation (specifically the [EOS] token) in CLIP. Conversely, tokens with high similarity often correspond to background or global context. This is attributed to CLIP's contrastive pretraining, which aligns global embeddings but leaves localized object details less aligned with the global text vector.
2. Text Attention Sink: The analysis reveals that the [EOS] token in CLIP exhibits a strong "attention sink" bias, directing the majority of its attention toward the [SOS] (start-of-sentence) token rather than the semantic content of the referring expression. This lack of rich semantic focus in the [EOS] token exacerbates the similarity reversal.

Based on these insights, LiteLVLM employs a two-stage selection process:

• Similarity-Aware Selection: Instead of retaining high-similarity tokens, LiteLVLM selects visual tokens with the lowest similarity to the [EOS] token. These tokens are empirically found to be spatially aligned with the referent regions, preserving crucial local cues for grounding.
• Context-Aware Recovery: To prevent the loss of global context and background information necessary for clear foreground-background separation, the method recovers additional tokens based on their contribution to the [CLS] token (measured via attention-weighted value vectors).
• Adaptive Selection: An adaptive strategy dynamically balances the budget between similarity-aware (low-similarity) and context-aware tokens based on the input text, ensuring optimal performance across diverse referring expressions.

Key Contributions

1. Identification of a Critical Flaw: The authors demonstrate that existing text-agnostic pruning methods and those relying on high visual-text similarity perform poorly on pixel grounding because they discard the very tokens needed to identify the referent.
2. CLIP Analysis: The paper provides a novel analysis of CLIP, uncovering the visual-text similarity reversal and the [EOS] token's attention sink bias, explaining why standard pruning heuristics fail for grounding tasks.
3. LiteLVLM: The proposal of the first training-free token pruning method specifically tailored for pixel grounding. It leverages reversed similarity rankings to retain referent-specific tokens while recovering context, requiring no fine-tuning.

Experimental Results
Extensive experiments were conducted on image and video benchmarks, including RefCOCO, RefCOCO+, RefCOCOg, Ref-DAVIS-17, and Refer-YouTube-VOS.

• Performance Retention: LiteLVLM significantly outperforms state-of-the-art methods (e.g., TRIM, LLaVA-PruMerge, VisPruner). On the RefCOCO+ test set, it retains 90.3% of the original performance while pruning 66.7% of the visual tokens (reducing 576 to 192 tokens). Even with an 88.9% reduction (64 tokens), it maintains a 79.2% relative performance score, surpassing competitors by over 10%.
• Video Grounding: In video object segmentation, LiteLVLM preserves 99.5% of the original performance with a 65.9% token reduction, demonstrating robustness to temporal redundancy.
• Efficiency: The method achieves a 22% inference speedup and a 2.3× reduction in memory overhead (activation storage) compared to the vanilla model. It also reduces FLOPs by over 2×.
• Generalization: The approach generalizes effectively to different LVLM architectures (e.g., Qwen2.5-VL) and vision encoders (e.g., SigLIP-2), confirming it is not tightly coupled to the specific CLIP implementation used in the initial analysis.

Significance and Claims
The paper claims that LiteLVLM offers a practical, cost-effective solution for deploying LVLMs on resource-constrained devices (edge devices, limited servers) without compromising pixel-level grounding accuracy. By reversing the standard intuition of token importance in CLIP, the method enables efficient inference while maintaining high fidelity in segmentation tasks. The authors emphasize that this is the first work to extend token pruning to pixel grounding across both image and video modalities, achieving state-of-the-art results without any training or fine-tuning.