Technical Summary: COAST – Contrastive Adaptive Semantic Token Pruning for Vision-Language Models

1. Problem Statement: Visual Aphasia in Token Pruning

Large Vision-Language Models (LVLMs) achieve strong multimodal reasoning by processing dense visual token sequences. However, the quadratic complexity of decoder self-attention necessitates visual token reduction to improve inference efficiency. Existing training-free pruning methods (e.g., FastV, LLaVA-PruMerge) typically rely on scalar attention scores from shallow layers to rank and discard "low-attention" tokens.

The authors identify a critical failure mode in this approach, termed Visual Aphasia. This phenomenon occurs when premature pruning based on shallow attention removes tokens that are initially low-scoring but become essential in deeper layers for resolving secondary objects, spatial relations, and contextual cues. When these tokens are discarded, the model loses visual grounding and falls back on language priors, leading to hallucinations (e.g., answering "The Camel Cafe" when asked about a small sign in an image dominated by a camel). The paper argues that scalar saliency inherently underestimates contextually emerging visual evidence required for compositional reasoning.

2. Methodology: COAST

To address Visual Aphasia, the authors propose COAST (COntrastive Adaptive Semantic Token Pruning), a training-free framework that reframes visual compression as adaptive semantic routing. Instead of a single scalar decision, COAST separates the retention budget into two complementary roles: semantic evidence (query-aligned) and spatial context (complementary).

The framework operates in two main stages at scheduled pruning layers:

A. Entropy-Driven Dynamic Budgeting

COAST reuses native cross-modal attention to estimate the dispersion of visual evidence.

1. Attention Entropy Calculation: It computes the global text-to-image attention entropy ($H$) from the preceding layer. Low entropy indicates concentrated attention on specific regions, while high entropy suggests dispersed evidence requiring broader context.
2. Adaptive Split: The remaining token budget (after reserving query-specific anchors) is split into two parts based on $H$:
   • $n_1$ (Semantic Evidence): Tokens aligned with the query.
   • $n_2$ (Complementary Spatial Context): Tokens providing scene-level context.
   • The split is adaptive: as attention entropy increases, the budget shifts toward preserving more spatial context ($n_2$), ensuring that dispersed cues are not lost.

B. Anchor–Reference Contrastive Semantic Routing

Once the budget is allocated, COAST scores candidate tokens using a contrastive mechanism rather than a single scalar score.

1. Set Construction:
   • Anchors ($A$): Selected via last-token attention (high relevance to the query).
   • References ($R$): Selected via global attention as the lowest-scoring tokens (low-salience context).
   • Candidates ($C$): The remaining non-anchor tokens.
2. Contrastive Scoring: For each candidate $c_i$, a score is computed by contrasting its similarity to the anchors against its similarity to the references:
   $$\text{Score}(c_i) = \text{Sim}_A(c_i) - \text{Sim}_R(c_i)$$
   Where $\text{Sim}_A$ is the maximum cosine similarity to any anchor, and $\text{Sim}_R$ is the average cosine similarity to the reference set.
3. Two-Tail Retention:
   • Top-$n_1$: Candidates with the highest scores are retained as anchor-aligned evidence.
   • Bottom-$n_2$: Candidates with the lowest scores are retained as complementary spatial context. These tokens are distinct from the anchors but are crucial for maintaining scene coverage and spatial relationships that scalar pruning would discard.
4. Ordered Recomposition: The retained tokens (Anchors + Top-$n_1$ + Bottom-$n_2$) are sorted by their original sequence indices to preserve spatial layout for subsequent decoder layers.

3. Key Contributions

• Identification of Visual Aphasia: The paper defines and demonstrates a specific failure mode where aggressive early pruning causes LVLMs to lose visual grounding, relying instead on language priors.
• COAST Framework: A training-free pruning method that treats compression as adaptive semantic routing, utilizing entropy to balance semantic evidence and spatial context.
• Contrastive Routing Mechanism: A novel scoring strategy that preserves both high-relevance anchors and low-salience complementary context, avoiding the "attention collapse" to the most salient object.
• Comprehensive Evaluation: Extensive experiments across seven benchmarks (MME, MMBench, AI2D, GQA, VizWiz, POPE, ScienceQA-IMG) and multiple LVLM families (LLaVA-1.5, LLaVA-NeXT, LLaVA-OneVision, Qwen2.5-VL).

4. Experimental Results

Across seven benchmarks, COAST demonstrates superior performance-efficiency trade-offs compared to strong baselines (FastV, PDrop, SparseVLM, V2Drop, MMTOK):

• Performance Retention: COAST reduces visual tokens by 77.8% (retaining 128 tokens) while retaining 98.64% of the original average performance. Under a 192-token budget, it incurs only a 0.21% performance drop, significantly outperforming baselines (e.g., SparseVLM at 1.61% drop).
• Latency Speedup: COAST achieves a 2.15× latency speedup. Notably, it achieves lower latency than baselines with comparable FLOPs reductions due to efficient routing and ordered token recomposition.
• Generalization: The method consistently outperforms baselines across different model scales (0.5B to 13B) and architectures, showing robustness in both moderate and aggressive compression settings.
• Ablation Studies:
  • Two-Tail Retention: Removing either the anchor-aligned branch ($n_1$) or the complementary context branch ($n_2$) degrades performance, confirming the necessity of preserving both types of evidence.
  • Entropy-Driven Budgeting: Adaptive splitting based on attention entropy outperforms fixed-ratio splitting, particularly on benchmarks requiring complex reasoning (MME, GQA).
  • Contrastive Scoring: The anchor-reference contrastive score significantly outperforms scalar attention or MaxSim-only scoring.

5. Significance and Claims

The paper claims that adaptive semantic routing is a robust alternative to one-shot scalar pruning. By acknowledging that token importance is dynamic and context-dependent, COAST mitigates the "Visual Aphasia" failure mode, allowing LVLMs to maintain fine-grained visual grounding even under aggressive token reduction. The authors emphasize that their method requires no fine-tuning, adds minimal computational overhead, and is applicable across diverse LVLM architectures, offering a practical solution for deploying efficient, high-fidelity multimodal reasoning systems.