Technical Summary: Entropy Meets Importance (HIES)

1. Problem Statement

Transformer-based models, particularly Large Language Models (LLMs), have achieved state-of-the-art performance but suffer from significant computational and memory costs during inference due to their deep structures and numerous attention heads. While structured pruning methods, specifically Head Pruning, offer a solution by removing redundant attention heads, existing approaches face critical limitations:

• Reliance on Gradient-Based Scores (HIS): Current methods predominantly use Head Importance Scores (HIS), which quantify a head's contribution to the loss function via gradient sensitivity. While effective at moderate sparsity, HIS-based methods exhibit a "sharp-drop" in accuracy when pruning ratios become aggressive.
• Instability and Lack of Diversity: HIS fails to capture the diversity of attention patterns. It treats a head with concentrated attention (focusing on decisive tokens) and a head with diffuse attention similarly if their gradient magnitudes are comparable. This leads to the removal of functionally critical heads that stabilize the model, resulting in poor robustness to input perturbations and distribution shifts.
• Layer-Agnostic Limitations: Existing criteria often apply uniform pruning rules across all layers, ignoring evidence that different layers require distinct attention behaviors, leading to imbalanced pruning and performance fluctuations.

2. Methodology: Head Importance–Entropy Score (HIES)

The authors propose HIES (Head Importance–Entropy Score), a unified pruning criterion that integrates gradient-based importance with the structural distribution of attention.

Core Components

1. Head Importance Score (HIS):

   • Measures the first-order loss increase if a head is masked.
   • Formula: $HIS_h = \mathbb{E}_{x \sim \mathcal{D}} \left| \frac{\partial L(x)}{\partial m_h} \right|$, where $m_h$ is the mask variable for head $h$.
   • Captures the magnitude of a head's contribution to the loss but ignores attention distribution patterns.

2. Attention Entropy (AE):

   • Measures the concentration or dispersion of a head's attention weights across input tokens.
   • Formula: $AE_h = -\sum_{i=1}^n p_i^{(h)} \log p_i^{(h)}$.
   • Low entropy indicates concentrated attention (often on decisive tokens), while high entropy indicates diffuse attention. The authors hypothesize that low-entropy heads are crucial for stability and preventing "entropy collapse."

3. The HIES Metric:

   • HIES combines normalized HIS and AE into a single score:
     $$HIES_h = \alpha \cdot \widehat{HIS}_h + (1 - \alpha) \cdot (1 - \widehat{AE}_h)$$
   • Normalization: Both HIS and AE are min-max normalized to $[0, 1]$ to ensure comparability across different scales.
   • Weighting: $\alpha$ is a tunable hyperparameter. The term $(1 - \widehat{AE}_h)$ ensures that heads with lower entropy (more concentrated) receive higher scores, prioritizing their retention.

Theoretical Justification

The paper provides a risk-decomposition analysis showing that:

• HIS controls the loss-increase term (minimizing immediate accuracy degradation).
• AE bounds the generalization gap (minimizing instability and sensitivity to distribution shifts).
• Orthogonality: The gradients of HIS and AE are shown to be orthogonal in expectation, meaning they capture complementary signals: magnitude of contribution vs. dispersion of attention. Minimizing the composite risk bound via HIES theoretically guarantees better stability and accuracy preservation.

3. Key Contributions

• Novel Pruning Criterion: Introduction of HIES, which unifies gradient-based importance with attention entropy to address the "sharp-drop" phenomenon in aggressive pruning.
• Stability Focus: Explicitly frames attention entropy as a proxy for inference-time stability and robustness, arguing that preserving low-entropy heads prevents structural diversity collapse.
• Theoretical Framework: Provides a formal risk bound linking head pruning to both loss increase and generalization gaps, justifying the orthogonality and complementarity of HIS and AE.
• Layer-Adaptive Pruning: Demonstrates that HIES enables more balanced pruning across layers compared to the bottom-up patterns often seen in HIS-only methods.

4. Experimental Results

The authors evaluated HIES on BERT-base, LLaMA-2-7B, ViT-Large, and LLaVA-1.5-7B across various NLP, vision, and multi-modal tasks (e.g., GLUE, HellaSwag, ImageNet, VizWiz-VQA).

• Model Quality:

  • On BERT-base, HIES improved average model quality by 8.21% over HIS across pruning ratios.
  • At a 50% pruning ratio, HIES achieved up to a 15.2% improvement in model quality compared to the best-performing baseline.
  • On LLaMA-2-7B, HIES improved accuracy by up to +10.54% over HIS.
  • On ViT-Large, at 20% sparsity, HIES achieved 49.82% higher accuracy than HIS, preventing the accuracy collapse observed in gradient-only methods.

• Stability:

  • HIES demonstrated up to a 2.04× improvement in stability (measured against the unpruned model) compared to HIS-only methods.
  • At 50% pruning on BERT, HIES showed an 8.65% stability gain over HIS.

• Generalizability:

  • The method consistently outperformed baselines (Random, L2-Norm, HIS, Attention Deficit, LLM-Pruner, SliceGPT) across NLP, vision, and complex multi-modal reasoning tasks.
  • HIES successfully delayed the "sharp-drop" regime to more aggressive sparsity levels, maintaining performance where HIS failed.

5. Significance and Claims

The paper claims that HIES offers a more practical and robust solution for deploying Transformer models in resource-constrained environments. By preserving both accuracy and stability even under aggressive compression, HIES addresses the critical barrier of unpredictable performance fluctuations in real-world deployments.

The authors emphasize that their work provides a principled direction for stable and efficient pruning, moving beyond simple gradient sensitivity to incorporate structural diversity (entropy) as a core component of model compression. They assert that this approach is broadly applicable across different architectures (BERT, LLaMA, ViT, LLaVA) and modalities, offering a reliable pathway for deploying efficient LLMs without sacrificing robustness.