HAWK: Head Importance-Aware Visual Token Pruning in Multimodal Models

1. Problem Statement

Multimodal Large Language Models (MLLMs) encode visual inputs (images/videos) into sequences of visual tokens, which are processed alongside text tokens.

• The Bottleneck: Visual inputs are inherently dense and spatially redundant, often generating hundreds or thousands of tokens (e.g., 576 tokens for a 336×336 image, or >200k for long videos).
• The Consequence: Since the computational complexity of attention mechanisms scales quadratically with sequence length, the surge in visual tokens drastically increases inference time and memory overhead, making real-time or resource-constrained deployment impractical.
• Limitations of Existing Methods: Current visual token pruning strategies generally fall into three categories:
  1. Similarity-based: Remove redundant regions based on feature similarity but ignore user instructions (context-agnostic).
  2. Fine-tuning-based: Learnable pruning modules that require expensive end-to-end training and suffer from poor generalization.
  3. Attention-based: Estimate token importance using early-layer attention scores. Crucially, these methods assume all attention heads contribute equally to visual interpretation, simply averaging head weights.
• The Gap: The authors argue that different attention heads capture distinct visual semantics and play unique, non-uniform roles in visual processing. Treating them equally leads to the retention of redundant tokens and the accidental pruning of semantically critical ones.

2. Methodology: HAWK

HAWK (Head Importance-Aware Visual Token Pruning) is a training-free method designed to maximize the retention of crucial tokens by explicitly accounting for the varying importance of attention heads.

Core Components:

1. Static Visual Head Importance Weights ($W$):

   • Observation: Through ablation studies (systematically disabling individual heads), the authors found that masking different heads causes markedly different drops in model performance across datasets.
   • Calculation: HAWK performs a one-time offline analysis using benchmark datasets. It calculates the performance drop ($\Delta S$) caused by ablating each head $i$.
   • Weighting: The importance weight $w_i$ for each head is derived by averaging the normalized performance drops across all datasets. This creates a static vector representing the intrinsic contribution of each head to visual understanding.

2. Dynamic Text-Guided Attention Scores ($C$):

   • Mechanism: To adapt to specific user queries, HAWK uses a text-guided attention mechanism. It computes the relevance between visual tokens ($H_v$) and textual instructions ($H_t$).
   • Implementation: It utilizes the Query ($W_q$) and Key ($W_k$) projection matrices from the first attention layer of the LLM.
   • Positional Agnosticism: To ensure attention depends solely on semantic correspondence rather than position, the authors omit Rotary Position Embeddings (RoPE) during this specific relevance calculation.
   • Scoring: The relevance score for each visual token is obtained by averaging attention scores across all text tokens for each head.

3. Head Importance-Aware Pruning:

   • Integration: HAWK combines the static head weights ($W$) with the dynamic text-guided scores ($C$) to compute a final importance vector $I$ for each visual token $k$:
     $$I_k = \sum_{i=1}^{N_h} w_i \cdot c_{i,k}$$
   • Selection: Given a target pruning ratio, the top-$\tilde{M}$ tokens with the highest $I_k$ scores are retained, while the rest are discarded. The pruned set is then concatenated with text tokens for standard LLM processing.

3. Key Contributions

1. Discovery of Head Heterogeneity: The paper provides empirical evidence that attention heads in MLLMs play distinct, non-interchangeable roles in visual processing. Quantifying this importance is critical for effective pruning.
2. HAWK Framework: Proposes a novel, entirely training-free pruning method that integrates static head importance with dynamic text-guided attention. It requires only a single offline computation to determine head weights.
3. State-of-the-Art Performance: Demonstrates consistent superiority over existing baselines (FastV, DivPrune, CDPruner) across diverse models (Qwen2.5-VL, InternVL3) and tasks (Image VQA, Video Understanding).

4. Experimental Results

The authors evaluated HAWK on multiple benchmarks including MME, HallBench, TextVQA, ChartQA, AI2D, RealWorldQA, CCBench, OCRVQA, SQA-IMG, POPE, and video benchmarks (Video-MME, WorldSense).

• Accuracy Retention:
  • On Qwen2.5-VL-7B (Fixed Resolution): When pruning 80.2% of visual tokens, HAWK retains 96.0% of the original accuracy. This outperforms the second-best method (FastV) by 3.4 percentage points.
  • On InternVL3-8B: At an 80% pruning ratio, HAWK maintains 94.1% performance, surpassing baselines by 7.0 percentage points.
  • Video Understanding: At a 90% pruning ratio, HAWK retains 93.7% of original performance, significantly outperforming other methods.
• Efficiency Gains:
  • Latency: Reduces end-to-end inference latency to 74.4% of the original time (a 1.34x speedup at 80% pruning).
  • Memory: Significantly reduces GPU memory usage and KV cache size (e.g., reducing KV cache from 668 MB to 148 MB at 80% pruning).
• Robustness: Performs consistently well on both fixed-resolution and native-resolution (variable length) inputs, as well as long-form video tasks.

5. Significance

• Practical Deployment: HAWK addresses the critical barrier of high computational cost in MLLMs, making them viable for real-time and resource-constrained applications without requiring costly retraining.
• Paradigm Shift: It challenges the prevailing assumption in attention-based pruning that all heads are equal. By leveraging head-specific importance, it achieves a more precise and semantically aware pruning strategy.
• Generalizability: As a plug-and-play, training-free module, HAWK can be seamlessly applied to various existing MLLM architectures, offering an immediate path to efficiency improvements for the multimodal community.