💬 NLP

High-Layer Attention Pruning with Rescaling

This paper introduces High-Layer Attention Pruning with Rescaling (HARP), a novel training-free method that strategically removes attention heads from higher layers of large language models and applies adaptive rescaling to preserve representation magnitude, thereby achieving superior performance across diverse generation and discriminative tasks compared to existing structured pruning techniques.

Original authors: Songtao Liu, Peng Liu

Published 2026-08-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Songtao Liu, Peng Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot brain, a "Large Language Model" (LLM), that can write stories, solve math problems, and chat like a human. These brains are incredibly powerful, but they are also massive—like trying to carry a library in your backpack. Because they are so huge, they are slow to think and take up a lot of space, which makes them hard to use on regular phones or laptops. To fix this, scientists use a technique called "pruning." Think of pruning like trimming a bonsai tree: you carefully cut away the branches that aren't doing much work so the tree stays healthy but becomes smaller and faster.

Usually, when people trim these robot brains, they look at the "attention heads"—the tiny parts of the brain that help the model decide which words are important to each other. Imagine these heads as a team of detectives in a room, each looking at a different clue. The old way of pruning was to just pick a few detectives from every single room and kick them out, hoping the team still works. But this paper suggests that's not the smartest way to do it. It turns out that in the very top rooms of the building (the higher layers of the model), the detectives are often just repeating what the ones below them already said. They aren't adding much new value. The authors of this paper, Songtao Liu and Peng Liu from Penn State, wanted to see if we could get a better result by only trimming the detectives from the top floors and leaving the hard-working ones on the lower floors alone. They also figured out a clever trick to make sure the brain doesn't get confused when they remove those top-floor detectives.

The Big Idea: Cutting from the Top, Not the Middle

The researchers proposed a new method called HARP (High-layer Attention Rescaled Pruning). Their main discovery is that the "higher layers" of a Large Language Model are actually less important than the lower ones. To understand why, imagine a relay race. The runners at the start (the lower layers) are doing the heavy lifting, figuring out the basic shape of the words and sentences. As the baton moves up the track to the higher layers, the runners are just passing it along. By the time the baton reaches the very top, the runners are so synchronized that they are all doing the exact same thing. If you remove a runner from the top of the track, the race doesn't really change because they were just copying the person below them anyway.

The authors found that if you prune (cut out) the attention heads in these top layers, the model actually works better than if you cut them out randomly from everywhere. They tested this on several famous models like LLaMA3.1-8B, Mistral-7B, Qwen2-7B, and Gemma2-9B. In their experiments, they removed specific attention heads from the highest 8 layers of the models.

The "Volume Knob" Trick

There was a catch, though. When you remove those top-layer detectives, the signal the brain sends out gets a little quieter or louder than before. It's like if you removed a speaker from a stereo system; the music might still play, but the volume would be off, and the sound could get distorted. To fix this, the authors introduced a "rescaling parameter." Think of this as a volume knob or a dimmer switch. After they cut the top layers, they turned this knob to adjust the size of the signals, making sure the remaining parts of the brain could still hear each other clearly. They didn't just guess the right setting; they used a smart search method to find the perfect volume level for each layer, testing different numbers until the model sounded its best.

What They Found

The results were pretty impressive. When they tested their HARP method on 27 different tasks—ranging from answering trivia questions to solving complex math problems and summarizing long documents—it consistently beat the other popular pruning methods.

Generation Tasks: This is where the model has to write new things, like solving a math problem step-by-step. Here, HARP shined the most. For example, on the LLaMA3.1-8B model, HARP got an average score of 31.10% on generation tasks, while the next best method only got 20.58%. That's a huge jump!
Long Context: The paper also looked at how well the models handled very long stories or documents (up to 65,536 tokens). Because the attention mechanism is the part that gets slowest with long text, cutting the top layers made the model significantly faster. They found that for a sequence of 65,536 tokens, their pruned model was 16.7% faster than the original, unpruned model, while still keeping the quality high.
Discriminative Tasks: These are tasks where the model just picks the right answer from a list (like a multiple-choice quiz). Here, the improvement was smaller but still positive. HARP performed as well as or better than the other methods, showing that cutting the top layers didn't break the model's ability to understand facts.

Why This Matters

The authors suggest that this method is a game-changer for making AI faster and cheaper to run without needing to retrain the whole model from scratch. By realizing that the top floors of the AI building are the most "redundant" (doing the least unique work), they can trim the fat without hurting the muscle. The "rescaling" trick ensures that the model doesn't get dizzy when you remove those parts.

In short, the paper argues that we shouldn't treat all parts of an AI brain the same. Just like a skyscraper, the top floors might be less critical for the structure than the foundation. By carefully trimming the top and adjusting the volume, we can make these giant AI brains smaller, faster, and ready for everyday use. The code for this method is available for anyone to try, and the authors hope it will help make long-context AI (like reading whole books in seconds) much more practical for the real world.

Technical Summary: High-Layer Attention Pruning with Rescaling (HARP)

Problem Statement

Large Language Models (LLMs) achieve state-of-the-art performance but suffer from high inference latency and memory costs, particularly in long-context scenarios. While structured pruning offers a training-free solution to reduce parameters, existing methods often employ heuristic metrics that indiscriminately remove attention heads across all network layers. These approaches fail to account for the positional redundancy of attention heads within the architecture. Specifically, current methods do not distinguish whether attention heads in lower layers (which process diverse token representations) are more critical than those in higher layers (where representations may have become highly aligned). Furthermore, removing attention heads alters the magnitude of token representations because the aggregation step (weighted averaging) is bypassed, potentially destabilizing the model's output scale.

Methodology

The authors propose HARP (High-layer Attention Rescaled Pruning), a training-free structured pruning algorithm designed to address these issues through two core mechanisms:

1. High-Layer Attention Pruning

HARP operates on the theoretical and empirical premise that attention heads in the upper layers of LLMs are less important than those in lower layers.

Theoretical Basis: As tokens propagate through the network, their hidden representations tend to become increasingly similar (over-smoothing). When the average pairwise cosine similarity of token representations approaches 1, the queries and keys become nearly parallel. Consequently, the softmax attention weights become uniform, causing the attention head to degenerate into a simple averaging operator that cannot distinguish between tokens.
Empirical Basis: Metrics such as Block Importance (BI) scores indicate that parameters in lower layers generally have a higher impact on model performance than those in higher layers.
Implementation: HARP selectively prunes the Query ( $Q$ ) and Key ( $K$ ) parameters of attention heads in the highest layers of the model. By removing $Q$ and $K$ , the self-attention mechanism is effectively bypassed for these layers. The layer then only applies the Value ( $V$ ) and Output ( $O$ ) projections, reducing the computational complexity from $O(N^2d)$ to $O(Nd)$ for the pruned layers.

2. Adaptive Rescaling

Pruning the attention aggregation step alters the scale (magnitude) of the hidden state representations, as the weighted averaging operation is removed.

Problem: Without the aggregation step, the Frobenius norm of the hidden states changes, potentially leading to performance degradation.
Solution: HARP introduces a non-differentiable rescaling parameter, $\alpha$ , into the residual connection of the pruned layers. The output is formulated as $H = H + \alpha H W_V W_O$ .
Optimization: Since $\alpha$ is a non-differentiable hyperparameter rather than a trainable parameter, the authors employ a greedy, top-down search strategy. Starting from the topmost pruned layer, they search for the optimal $\alpha$ value (from a discrete set $\{0, 0.1, \dots, 1.0\}$ ) by minimizing the Perplexity (PPL) on a held-out corpus. This process is repeated layer-by-layer down the network.

Key Contributions

Novel Pruning Strategy: The introduction of HARP, which targets attention heads specifically in the higher layers of LLMs, based on the observation that these heads are less informative due to representation over-smoothing.
Rescaling Mechanism: The proposal of an adaptive rescaling parameter to calibrate token representation magnitudes post-pruning, counteracting the scale shift caused by bypassing the attention aggregation step.
Comprehensive Evaluation: Extensive experiments across four diverse LLM architectures (LLaMA3.1-8B, Mistral-7B-v0.3, Qwen2-7B, Gemma2-9B) and 27 downstream tasks, covering both generation and discriminative benchmarks.
Efficiency Gains: Demonstration of significant inference speedups, particularly in long-context tasks, by reducing the quadratic complexity of the attention mechanism in pruned layers.

Experimental Results

The authors evaluated HARP against state-of-the-art baselines including Magnitude Pruning (Mag-sp), LLM-Pruner, FLAP, and Wanda.

Generation Tasks: HARP consistently outperformed all baselines. On LLaMA3.1-8B, HARP achieved an average score of 31.10% on generation tasks (TriviaQA, NaturalQuestions, GSM8K, MATH-hard, BBH), significantly surpassing FLAP (15.05%) and Wanda-sp (12.61%). The improvement was particularly notable in tasks requiring long responses (e.g., GSM8K), where error propagation is a major challenge for pruned models.
Discriminative Tasks: HARP achieved the highest average performance on LLaMA3.1-8B, Mistral-7B-v0.3, and Qwen2-7B, and performed on par with the best baseline on Gemma2-9B.
Long-Context Modeling: On the LongBench benchmark, HARP demonstrated superior robustness in multi-document QA and summarization tasks, preserving long-range dependencies better than other methods.
Ablation Studies:
- Layer Selection: Pruning the top 8 layers (Top-8) yielded significantly better results than pruning the bottom 8 layers (Bottom-8) or using Hessian/Similarity-based metrics to select layers.
- Rescaling: The inclusion of the searched $\alpha$ parameter resulted in substantial performance gains, particularly on generation tasks, confirming the necessity of magnitude calibration.
Efficiency: For a 65,536-token sequence, HARP achieved a 16.7% speedup in inference time with only a 3.3% parameter reduction, highlighting the efficiency of targeting attention heads in long-context scenarios.

Significance and Claims

The paper claims that HARP provides a practical, training-free method to compress LLMs without requiring architectural changes or retraining from scratch. By strategically targeting the redundancy in higher layers and correcting the resulting scale shifts, HARP facilitates the deployment of LLMs in resource-constrained environments and enables faster serving for interactive applications. The authors emphasize that their approach makes long-context reasoning more affordable in real-world systems.

The study acknowledges limitations, noting that it focuses primarily on attention-head pruning in higher layers, leaving other components underexplored. Future work is suggested to extend HARP to larger and multimodal models and to combine it with quantization or KV-cache compression techniques.