Not All Attention is Needed: Parameter and Computation Efficient Transfer Learning for Multi-modal Large Language Models

1. Problem Statement

Multi-modal Large Language Models (MLLMs) have achieved significant success but face two major bottlenecks when adapting to downstream tasks:

• High Computational & Parameter Overhead: Fine-tuning MLLMs for specific applications requires massive computational resources and storage, making them inaccessible for many users.
• Inefficiency of Existing Methods:
  • Parameter-Efficient Transfer Learning (PETL): Methods like Adapters and LoRA reduce trainable parameters but often introduce latency during inference because they add parallel branches (skip connections) that cannot be easily merged into the original architecture without re-parameterization.
  • Coarse-Grained Skipping: Previous attempts to reduce computation (e.g., Dynamic Architecture Skipping or DAS) skip entire Transformer layers. This is suboptimal because MLLM components serve different roles: Multi-Head Attention (MHA) handles dependency modeling, while Feed-Forward Networks (FFN) enhance capacity. The paper observes that MHA is often more redundant for specific tasks than FFN, yet skipping entire layers forces the removal of both, potentially harming performance.
  • Latency vs. Efficiency: Simply removing layers with adapters still incurs inference overhead due to the adapter's execution path.

2. Methodology: Effective Attention Skipping (EAS)

The authors propose EAS, a framework designed to achieve true parameter and computation efficiency by skipping redundant MHA modules while maintaining performance and zero extra inference latency.

A. Core Insight: Granular Redundancy

The authors empirically demonstrate that:

1. MHA Redundancy: MLLMs contain significant redundancy in their Multi-Head Attention modules for downstream tasks. Skipping MHA layers reduces computation significantly (MHA consumes ~35% more time than FFN despite having fewer parameters) with minimal performance loss.
2. FFN Sensitivity: Skipping FFN layers causes a drastic drop in performance.
3. Conclusion: "Not all attention is needed." The optimal strategy is to skip MHAs specifically, rather than entire Transformer layers.

B. Propagation-of-Information Adapter (PIA)

To replace skipped MHAs without introducing inference latency, the authors design a novel module called PIA:

• Architecture: PIA uses a bottleneck structure with a multi-path design. It projects input features into low-dimensional spaces and incorporates global information propagation via average pooling. This allows the module to exchange information across tokens, mimicking the dependency modeling role of MHA.
• Re-parameterization (Zero-Latency): This is the key innovation. During training, PIA acts as a lightweight adapter. However, after training, its weights can be mathematically re-parameterized and merged directly into the existing Feed-Forward Network (FFN) weights.
  • The PIA is embedded in the residual connection of the FFN.
  • Through linear algebraic simplification (merging down-sampling, up-sampling, and routing weights), PIA is converted into a single linear layer that is added to the FFN's projection matrices.
  • Result: During inference, the PIA disappears as a separate module, incurring zero extra latency and zero extra FLOPs compared to the original model structure.

C. Redundancy Evaluation via K-Armed Bandit

To determine which MHAs to skip, EAS employs a K-Armed Bandit algorithm:

• It treats the selection of redundant MHA modules as a combinatorial optimization problem.
• The algorithm samples sub-networks, evaluates their performance, and updates an "action preference" score for each MHA module.
• Modules with low scores (high redundancy) are identified for skipping, while high-scoring modules are retained. This allows for an automated, task-specific selection of which attention heads/layers to remove.

3. Key Contributions

1. Effective Attention Skipping (EAS): A novel tuning method that targets MHA redundancy specifically, offering a more granular approach than layer-skipping methods.
2. Propagation-of-Information Adapter (PIA): A new adapter design that not only facilitates information exchange for skipped attention but can be fully re-parameterized into the model's FFN, achieving true model acceleration with no inference overhead.
3. Comprehensive Validation: The method is validated on three diverse MLLM architectures (LLaVA, LaVIN, METER) across six benchmarks (ScienceQA, Slake, AID, VQA2.0, NLVR2, Flickr30K).

4. Experimental Results

The paper reports extensive experiments showing EAS outperforms state-of-the-art PETL and PCETL methods:

• Performance vs. Speed Trade-off:

  • LLaVA on ScienceQA: EAS (skipping 12 MHAs) achieves 96.2% accuracy (comparable to full tuning) while accelerating inference by ~20% (1.20x speedup) and reducing updated parameters to <1%.
  • LaVIN on ScienceQA: EAS improves inference speed by 2.18x (from 3.70s to 1.69s per sample) while actually improving accuracy from 89.4% to 90.0% compared to the baseline LaVIN.
  • Out-of-Domain Generalization: On medical (Slake) and remote sensing (AID) datasets, EAS maintains high accuracy (e.g., 90.2% on Slake) while significantly reducing FLOPs compared to full fine-tuning and standard adapters.

• Comparison with DAS (Dynamic Architecture Skipping):

  • EAS consistently outperforms DAS in both speed and accuracy. For the same number of skipped layers, EAS is 44.4% to 51.6% faster because DAS retains adapter latency, whereas EAS merges weights.
  • EAS can skip more layers (up to 12) without performance degradation, whereas DAS performance drops significantly when skipping more than 4-6 layers.

• Ablation Studies:

  • MHA vs. FFN: Skipping MHA yields better efficiency/performance trade-offs than skipping FFN.
  • PIA Design: The multi-path design with global information propagation is crucial; removing it causes a ~3.3% accuracy drop.
  • Re-parameterization: Merging PIA into FFN is essential for achieving the reported speedups.

5. Significance

This paper addresses a critical gap in the deployment of MLLMs: the trade-off between efficiency and latency.

• True Acceleration: Unlike previous "efficient" methods that trade parameter count for inference latency, EAS achieves both parameter efficiency and actual inference speedup.
• Scalability: By proving that MHA modules are highly redundant in MLLMs, the paper suggests that future models could be designed with even fewer attention layers or that existing models can be significantly compressed for edge deployment without retraining from scratch.
• Practicality: The re-parameterization technique allows the final model to be deployed as a standard Transformer without any custom inference kernels, making it highly practical for real-world applications.

In summary, EAS demonstrates that "not all attention is needed," providing a robust, latency-free framework for adapting large multi-modal models to specific tasks.