Technical Summary: CAPA – Contribution-Aware Pruning and FFN Approximation for Efficient Large Vision-Language Models

1. Problem Statement

Large Vision-Language Models (LVLMs) face significant computational bottlenecks due to the high cost of processing thousands of visual tokens, particularly when handling high-resolution images. While visual token pruning has emerged as a primary acceleration strategy, existing methods rely heavily on raw attention scores or heuristic metrics derived from them to identify unimportant tokens. The authors argue that attention scores are an imperfect proxy for actual token importance because they ignore the magnitude of the associated value vectors.

Furthermore, current approaches often treat visual and text tokens uniformly. However, recent evidence suggests that LVLM backbones process these modalities differently. Specifically, redundancy in Feed-Forward Networks (FFNs) regarding visual tokens remains largely unexplored. Prior work in the LLM domain has explored FFN approximation (e.g., sparse approximation, MoE), but these techniques have not been extended to the specific redundancy patterns of vision tokens in LVLMs.

2. Empirical Analysis and Key Observations

The authors conducted empirical analyses on LVLMs (LLaVA-1.5, Qwen2.5-VL, InternVL3) to deconstruct visual token behaviors, leading to two critical observations:

1. Heterogeneity of Visual Attention Sinks:

   • Traditional "attention sinks" (tokens with disproportionately high attention mass) are not functionally uniform.
   • Using a new metric, Attention Contribution, the authors distinguish between two types of sinks:
     • Type I (Probability Dumps): Tokens with high attention weights but low value vector magnitudes. These act as passive receptacles for excess probability mass and contribute negligibly to the residual stream. They are safe to prune.
     • Type II (Structural Anchors): Tokens with both high attention weights and massive value vector magnitudes. These are critical for maintaining model performance; their removal causes representational collapse.
   • Conclusion: Relying solely on attention scores fails to distinguish between these two types, leading to the accidental pruning of essential tokens.

2. Modality-Dependent FFN Redundancy:

   • The authors measured the linearity of FFN transformations by calculating the cosine similarity between the input hidden state $x$ and the residual output $x + \text{FFN}(x)$.
   • Findings: Image tokens exhibit near-identity mapping (similarity > 0.96) across the majority of intermediate layers, indicating that the FFN transformations are effectively linear and redundant for visual data. In contrast, text tokens show significant non-linearity in early and middle layers.
   • Conclusion: The expensive $O(d^2)$ FFN computations for visual tokens in intermediate layers can be replaced with lightweight linear approximations without compromising representational integrity.

3. Methodology: CAPA Framework

Based on these findings, the authors propose CAPA (Contribution-Aware Pruning and FFN Approximation), a dual-strategy framework:

A. Contribution-Aware Pruning

Instead of using raw attention scores, CAPA computes an Attention Contribution Score ($C_i$) for each visual token $i$. This score weights the attention probability by the magnitude of the value vector projected by the output matrix.

• Formula: $C_i = \left\| \sum_{h=1}^{H} A_{i,h} (x_i W_{V,h}) W_{O,h} \right\|_2$
• Mechanism: During generation, the system dynamically selects the top-$k$ visual tokens with the highest contribution scores to retain in the key-value cache, pruning those with negligible contribution. This ensures that "Structural Anchors" are preserved while "Probability Dumps" are removed.

B. Redundancy Identification & FFN Approximation

For layers identified as having high redundancy (where input-output cosine similarity exceeds a threshold $\eta$), CAPA replaces the standard dense FFN with a lightweight Hadamard product.

• Approximation: The residual block $y^{(l)} = x^{(l)} + \text{FFN}(x^{(l)})$ is approximated as $\hat{y}^{(l)} = x^{(l)} \odot \alpha$, where $\alpha$ is a learned element-wise scaling vector.
• Complexity Reduction: This reduces the computational complexity of the layer from $O(d^2)$ to $O(d)$.
• Optimization: The optimal scaling vector $\alpha$ is derived analytically using an Ordinary Least Squares (OLS) solution on a calibration set, avoiding iterative gradient descent.

4. Experimental Results

The framework was evaluated on three LVLM backbones (LLaVA-1.5-7B, Qwen2.5-VL-7B, InternVL3-8B) across six benchmarks (VQAv2, MMBench, MMVet, TextVQA, SEED-Bench, MMMU).

• Performance vs. Baselines: CAPA consistently outperformed existing pruning baselines (FastV, Feather, Uniform pruning) across all benchmarks. Notably, at critical "transition layers" where pruning is most sensitive, CAPA maintained performance close to the unpruned Vanilla model, whereas baselines suffered significant degradation.
• Robustness: CAPA demonstrated superior robustness at high sparsity levels. While baselines experienced precipitous performance drops beyond 50% pruning, CAPA maintained near-vanilla accuracy even at 75% pruning.
• Efficiency Gains:
  • Token Pruning: Reduced the quadratic cost associated with sequence length.
  • FFN Approximation: Eliminated the dominant $O(d^2)$ cost in redundant layers.
  • Total Impact: The combined approach achieved a 78% reduction in total FLOPs while preserving task performance.
• Ablation Studies:
  • Logit Divergence: Pruning based on Attention Contribution resulted in lower Hellinger distances (closer to the vanilla model's output distribution) compared to attention-score-based pruning, especially in long-form generation.
  • FFN Replacement: Simply skipping FFN layers caused performance drops, but replacing them with the Hadamard product successfully recovered the majority of the lost performance, validating the linearity hypothesis.

5. Significance and Claims

The paper claims that CAPA addresses a critical gap in LVLM optimization by recognizing that visual tokens behave differently from text tokens in both attention mechanisms and FFN processing.

• Shift in Pruning Paradigm: The authors argue that attention scores are insufficient for token selection. By introducing Attention Contribution, they provide a more faithful signal for identifying which visual tokens are functionally necessary.
• Modality-Specific Optimization: The work highlights that efficiency gains in LVLMs require distinct strategies for visual and text modalities. Specifically, the identification of linear redundancy in visual FFNs allows for architectural simplification that is not applicable to text processing.
• Efficiency-Performance Trade-off: CAPA achieves a favorable trade-off, delivering substantial inference acceleration (78% FLOP reduction) without compromising model robustness or task performance, outperforming existing token-pruning-only baselines.

6. Limitations

The authors acknowledge that CAPA is a post-hoc strategy operating on fixed, pre-trained backbones. Consequently, it cannot fully exploit the benefits of jointly learned token selection and model computation. Future work is suggested in the direction of:

• Designing learned controllers for adaptive token retention.
• Developing vision-language backbones with native, efficient FFN modules tailored for visual tokens.
• Evaluating these strategies across diverse distribution shifts and investigating the interpretability of learned policies.