Identifying and Evaluating Inactive Heads in Pretrained LLMs

This paper proposes and evaluates 12 score functions to identify inactive attention heads in pretrained LLMs, revealing that over 12% of heads are redundant and can be ablated with minimal performance loss, while demonstrating that methods relying solely on attention weights or first-token sinks significantly underestimate this prevalence.

The Gist

Imagine a Large Language Model (LLM) like a massive, bustling newsroom. Inside this newsroom, there are hundreds of specialized editors (called attention heads) working together to write an article. Each editor looks at the incoming story (the input text) and decides which parts are important to focus on.

For a long time, researchers thought these editors were all working hard. But recently, they noticed something strange: some editors seemed to be staring blankly at the very first word of the story, ignoring everything else. This is called an "attention sink."

The big question this paper asks is: Are these editors actually "sleeping on the job," or are they just doing a weird kind of work? And if they are sleeping, can we fire them (or turn them off) to make the newsroom run faster without ruining the quality of the articles?

Here is the breakdown of their discovery, using simple analogies:

1. The Old Way of Checking (The "Gaze" Test)

Previously, researchers tried to find "lazy" editors by only looking at where they were staring.

• The Logic: If an editor is staring 90% of the time at the first word (the "sink"), they must be doing nothing useful.
• The Flaw: This is like judging a chef by only looking at which pot they are staring at. Maybe they are staring at the first pot, but they are actually stirring it with a tiny spoon, or maybe they are staring at the pot but the pot is empty!
• The Result: Using this old method, they found that only about 5% of the editors were "lazy."

2. The New Way of Checking (The "Output" Test)

The authors of this paper said, "Wait a minute. Let's not just look at where they are staring. Let's look at what they actually produce."

• They created 12 different tests (score functions) to measure an editor's activity.
  • Test A: Are they staring at the first word? (The old way).
  • Test B: Are they staring at a few specific words?
  • Test C: Are the ingredients they are holding (value vectors) empty?
  • Test D (The Winner): What is the final output? If an editor takes all the information, processes it, and produces a result that is basically zero (a whisper instead of a shout), then that editor is truly inactive.

3. The Big Discovery: The Newsroom is Half Empty!

When they used the new "Output Test" (specifically measuring the Average Head Output Norm), they found a shocking truth:

• More than 12% of the editors are actually asleep on the job.
• In some models, it's even higher!
• The Proof: They tried "firing" (zeroing out) these lazy editors. They turned them off completely while the model was answering questions.
• The Result: The model's performance barely dropped (less than 1% difference). The newsroom kept producing perfect articles even with a significant chunk of the staff turned off.

4. Why the Old Method Failed

The paper explains that the old method (looking only at "staring patterns") was misleading.

• The Analogy: Imagine an editor who is staring at the first word, but they are actually holding a very heavy, useless rock (a large value vector) and dropping it. The old method would say, "Oh, they are staring at the first word, so they are a 'sink'!" But the new method says, "Wait, they dropped a heavy rock, so they did something."
• Conversely, some editors might be staring at the first word, but they are holding an empty hand (zero value). The old method might miss them if the "staring" isn't strong enough, but the new method catches them because their output is zero.

5. What This Means for the Future

• Efficiency: If we know which editors are sleeping, we can turn them off dynamically. This could make AI run faster and use less battery power on your phone.
• Stability: The researchers found that once a model is trained, these "sleeping" patterns don't change much, even if you tweak the model later (fine-tuning). It's like the newsroom has a permanent shift schedule where certain people always take a nap.
• Scale: Interestingly, as models get bigger (from small to huge), the behavior of these editors stays surprisingly similar until the models get massive.

The Takeaway

Think of AI models as giant teams where some members are effectively doing nothing. For a long time, we only looked at who they were looking at to decide if they were working. This paper taught us to look at what they actually contributed.

By switching our focus from "where they look" to "what they produce," we discovered that over 12% of the team is redundant. We can likely turn them off to make AI faster, cheaper, and more efficient, without losing any of the "intelligence" we love.

Technical Summary

1. Problem Statement

Large Language Models (LLMs) rely on the Transformer architecture, where multi-head attention mechanisms allow tokens to aggregate information from relevant parts of the input sequence. However, recent observations suggest that attention mechanisms can become "dormant" or "inactive," leading to computational redundancy.

• The Phenomenon: Previous work identified "attention sinks," where the first token (or specific delimiters) receives disproportionately high attention weights despite having limited semantic importance.
• The Limitation of Prior Work: Existing methods for identifying inactive heads rely almost exclusively on attention weights (e.g., measuring if a head attends heavily to the first token). The authors argue this is insufficient because a head's output is a convex combination of value vectors. A head could attend to multiple tokens with near-zero value states, or have low-value vectors overall, resulting in a near-zero output even if attention weights are not concentrated on a single "sink" token.
• The Core Question: How prevalent are truly inactive attention heads, and what is the most robust metric for identifying them across different model families and scales?

2. Methodology

The authors propose a comprehensive framework to evaluate 12 distinct score functions designed to measure different definitions of "inactivity."

A. Score Functions

The paper categorizes inactivity metrics into three components of the attention mechanism:

1. Attention Weights:
   • Avg Weight of First Token (AWFT): Measures if a head focuses on the first token (the traditional "sink" metric).
   • Avg Entropy of Query Distributions: Measures if attention is overly concentrated on a few tokens (low entropy).
2. Value Vectors:
   • First Token Value Vector Norm: Checks if the value vector of the first token is near-zero.
   • Avg Value Vector Norm: Checks if the average magnitude of value vectors in the head is near-zero.
3. Head Outputs:
   • Last Token Head Output Norm: Measures the output magnitude at the final position (critical for next-token prediction tasks).
   • Avg Head Output Norm (AHON): Measures the average magnitude of the output vector across all positions.

Note: The authors also evaluate "Layer Normalized" (LN) versions of these scores, where a head's score is normalized by the average score of other heads in the same layer to account for inter-layer variance.

B. Experimental Setup

• Models: 14 models across 3 families: Llama-3 (3B, 8B, Instruct variants), OLMo-2 (7B, SFT, DPO, Instruct variants), and Qwen2.5 (0.5B to 14B).
• Intervention Strategy: Instead of static pruning (permanently removing heads), the authors perform dynamic model intervention. During forward passes on the MMLU benchmark (and others like PIQA and WinoGrande), they zero out the outputs of heads identified as "inactive" by specific score functions.
• Evaluation: They measure the drop in model accuracy. A score function is considered effective if it identifies a large percentage of heads that can be zeroed out while maintaining accuracy within 1% of the baseline pretrained model.

3. Key Contributions

1. Systematic Evaluation of 12 Metrics: The paper moves beyond the single-metric approach of prior work, evaluating a diverse set of score functions covering attention weights, value vectors, and head outputs.
2. Discovery of the "Avg Head Output Norm" Metric: The authors identify that measuring the Average Head Output Norm (specifically the Layer Normalized version, AHON (LN)) is the most robust and model-agnostic method for identifying inactive heads.
3. Quantification of Redundancy: They demonstrate that >12% of attention heads in pretrained LLMs are truly inactive (can be zeroed out with <1% accuracy drop) on average.
4. Critique of Attention Sinks: The paper proves that relying solely on "Attention Sinks" (AWFT) significantly underestimates inactivity. AWFT fails to identify ~7% more inactive heads compared to output-based metrics and performs poorly on certain model families (e.g., OLMo-2).
5. Insights into Training Dynamics:
   • Finetuning: Score distributions show that finetuning (SFT, DPO, RLHF) causes minimal changes to attention behavior.
   • Scaling: Attention behavior is largely invariant to model scale until very large sizes (e.g., 14B parameters), where specialization begins to diverge.

4. Key Results

• Performance of Metrics:
  • AHON (LN) ranked 1st for 8 out of 14 models and in the top-3 for 13 out of 14 models.
  • AWFT (the prior standard) ranked 1st for only 2 models and performed worst on the OLMo-2 family.
  • Table 2 Results: On average, AWFT identified only 4.61% of heads as inactive. In contrast, the authors' best score functions identified 12.18% of heads as inactive. This means prior methods missed roughly 7% of the computationally redundant heads.
• Robustness: The AHON (LN) metric is stable across different datasets (MMLU, PIQA, WinoGrande) and model families, whereas AWFT is highly sensitive to the dataset and task.
• Intervention Impact: Zeroing out the heads identified by AHON (LN) allowed the models to maintain MMLU accuracy within 1% of the baseline, confirming these heads are functionally redundant in those contexts.
• Finetuning Stability: Wasserstein distance analysis of score distributions revealed that finetuned models (Instruct, SFT, DPO) retain nearly identical attention behavior to their base models.

5. Significance and Implications

• Theoretical Understanding: The paper challenges the prevailing view that "attention sinks" are the primary indicator of head inactivity. It establishes that head output magnitude is a more reliable signal of redundancy than attention weights alone.
• Efficiency and Compression:
  • Dynamic Pruning: The findings suggest that LLMs could dynamically skip computation for specific heads during inference, potentially reducing latency and energy consumption.
  • KV Cache Reduction: Identifying inactive heads could allow for the compression of Key/Value caches, as inactive heads contribute little to the final hidden state.
• Architecture Design: The results suggest that future Transformer architectures could be designed with sparsity in mind, potentially utilizing routers to select active heads dynamically (similar to Mixture-of-Experts) rather than computing all heads for every token.
• Limitations: The study focuses on single forward passes (log-likelihood evaluation). The authors note that dynamic pruning during autoregressive generation is complex, as zeroing an incorrect head early in generation could cascade into errors later in the sequence.

In conclusion, this work provides a rigorous, model-agnostic framework for identifying computational redundancy in LLMs, shifting the focus from attention weights to head outputs and revealing that a significant portion of attention heads in modern models are effectively inactive.