🤖 machine learning

Locality-Aware Redundancy Pruning for LLM Depth Compression

This paper proposes LoRP, a training-free, one-shot depth pruning framework that dynamically allocates layer removal based on a novel Representation Locality Score to effectively reduce redundancy in large language models while preserving performance.

Original authors: Vincent-Daniel Yun, Youngrae Kim, Woosang Lim, YoungJin Heo, Minkyu Kim, Sunwoo Lee

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Vincent-Daniel Yun, Youngrae Kim, Woosang Lim, YoungJin Heo, Minkyu Kim, Sunwoo Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, incredibly deep library of knowledge (a Large Language Model). This library has hundreds of floors (layers), and each floor contains a team of librarians (neurons) who help process information. The problem is that this library is huge, expensive to run, and slow to navigate. You want to remove some floors to make it faster, but you're terrified that if you remove the wrong ones, the library will stop making sense.

This paper introduces a new way to decide which floors to knock down, called LoRP (Locality-Aware Redundancy Pruning). Here is how it works, explained simply:

The Old Way: Guessing and Checking

Previously, people tried to shrink these libraries using two main strategies:

The "Local" Approach: They looked at each floor individually and said, "This floor seems weak, let's remove it." They assumed every floor was unique and independent.
The "Contiguous" Approach: They assumed that redundancy (unnecessary repetition) only happens in a specific block of floors, like floors 10 through 15. So, they would just chop out that whole chunk.

The Problem: The authors discovered that different libraries (AI models) have different "architectures." In some libraries, the extra, useless work is concentrated in one specific block. In others, the extra work is spread out evenly across the entire building. The old methods used a "one-size-fits-all" rule, which often resulted in cutting out important floors or leaving behind useless ones.

The New Way: LoRP (The Smart Architect)

LoRP is like a smart architect who walks through the library before making any cuts to see how the floors actually relate to each other. It doesn't need to retrain the library (it's "training-free"); it just observes how the librarians talk to each other.

Here is the step-by-step process:

1. The "Echo" Test (Measuring Similarity)

The architect sends a few sample sentences through the library and listens to how the "hidden thoughts" (representations) change from floor to floor.

If Floor 5 and Floor 6 are saying almost the exact same thing, they are redundant.
If Floor 5 is saying something totally different from Floor 6, they are unique.

2. The "Locality Score" (The RLS)

The architect calculates a score called the Representation Locality Score (RLS). Think of this as a "clumpiness meter":

High Score (Clumpy): The library has "echo chambers." Floors 10–20 are all repeating the same ideas. The redundancy is localized.
Low Score (Spread Out): The library has "diffuse echoes." The repetition is scattered all over the building, from the basement to the roof. The redundancy is globally distributed.

3. Grouping the Floors (Clustering)

Based on that score, the architect groups the floors into "neighborhoods" (clusters) of similar thinking.

If the library is "clumpy," the neighborhoods are tight groups of identical floors.
If the library is "spread out," the neighborhoods are more diverse.

4. The Two-Stage Cut (Pruning)

Now, the architect decides which floors to remove using a two-step strategy:

Step 1 (Coverage): First, they make sure to remove one floor from every single neighborhood. This ensures they don't accidentally wipe out an entire unique neighborhood.
Step 2 (The Cleanup): Then, they look at the remaining floors in each neighborhood. They keep removing the most "redundant" (most repetitive) floors until they hit their target size.

Why It Works Better

The paper tested this on several different types of AI models (like LLaMA, Mistral, and Qwen).

For "Clumpy" models: LoRP correctly identified that it could safely remove a big chunk of floors from one specific area.
For "Spread Out" models: LoRP correctly realized it needed to take a little bit from many different areas, rather than chopping off a big block.

The Result:
Because LoRP adapts to the specific shape of the library, it keeps the AI smarter and more accurate than the old "one-size-fits-all" methods. It preserves the library's ability to answer questions and understand language much better, even after removing a significant number of floors.

In a Nutshell

Think of it like editing a movie.

Old methods were like blindly cutting out every 10th scene or cutting out a whole 5-minute block because they assumed the movie was repetitive in a specific way.
LoRP is like a director who watches the whole movie first, sees exactly where the dialogue repeats, and then cuts the specific redundant lines while keeping the story flow intact, no matter how the movie was originally structured.

The paper concludes that understanding where and how information repeats in an AI is the key to shrinking it efficiently without breaking it.

Technical Summary: Locality-Aware Redundancy Pruning for LLM Depth Compression

1. Problem Statement

Large Language Models (LLMs) based on Transformer architectures suffer from extreme depth and overparameterization, leading to substantial computational and memory costs during inference. While layer-wise (depth) pruning is an effective strategy to reduce latency and memory by removing entire Transformer blocks, existing one-shot, training-free methods rely on flawed assumptions regarding redundancy.

Current approaches generally fall into two categories:

Local-importance based methods: Estimate redundancy using isolated layer scores (e.g., Block Influence).
Contiguous similarity based methods: Assume redundancy is concentrated within adjacent depth regions.

However, empirical analysis reveals that these assumptions are insufficient. Inter-layer redundancy is not uniform across architectures; it can be either localized (concentrated in specific depth regions) or globally distributed (spread across the entire network). Fixed pruning priors fail to account for these architecture-dependent structural differences, leading to suboptimal performance degradation.

2. Methodology: Locality-Aware Redundancy Pruning (LoRP)

The authors propose LoRP, a training-free, one-shot depth pruning framework that adapts to the specific representation geometry of each LLM. The workflow consists of five key stages:

2.1 Inter-Layer Similarity Measurement

Using a small calibration dataset (128 sequences from C4), LoRP collects input hidden-state activations for every Transformer block. It computes a pairwise inter-layer hidden-state cosine similarity matrix ( $S$ ), where each entry $S_{ij}$ represents the mean per-token cosine similarity between the input representations of layers $i$ and $j$ .

2.2 Representation Locality Score (RLS)

To quantify the global structure of redundancy, the authors introduce the Representation Locality Score (RLS).

First, the global off-diagonal similarity mean ( $\bar{S}_{off}$ ) is calculated across all layer pairs.
The RLS is defined as: $RLS(S) = -\log_2(\bar{S}_{off})$ .
Interpretation: A high RLS indicates rapid decay in similarity across depth, implying localized redundancy (redundancy is concentrated in specific regions). A low RLS indicates strong global similarity, implying distributed redundancy (redundancy is spread across the network).

2.3 Layer Clustering

Based on the similarity matrix $S$ and the RLS, layers are partitioned into representational clusters using spectral clustering on an affinity matrix $A = (S + 1)/2$ .

The granularity of clustering is guided by the RLS: higher RLS leads to coarser clustering (concentrated regions), while lower RLS encourages finer-grained partitioning to handle distributed redundancy.
This step groups layers that process similar residual-stream representations, regardless of their physical adjacency.

2.4 Two-Stage Redundancy-Aware Pruning

LoRP allocates the pruning budget ( $k$ layers to remove) via a two-stage procedure to balance coverage and redundancy:

Stage 1: Coverage-aware Initialization: To prevent collapsing pruning into a single cluster, the method selects the most redundant eligible layer from each cluster to form an initial candidate pool.
Stage 2: Residual Redundancy Allocation: The remaining budget is allocated iteratively. At each step, the algorithm identifies the cluster with the highest residual redundancy (mean pairwise similarity among remaining eligible layers) and prunes the most redundant layer within that cluster.

This dynamic allocation ensures that architectures with localized redundancy prune within concentrated regions, while those with distributed redundancy prune across multiple regions.

3. Key Contributions

The paper makes the following specific contributions:

Empirical Observation: Demonstrated that representation locality varies substantially across LLM families (e.g., Qwen models exhibit distributed similarity, while LLaMA/Mistral exhibit localized redundancy).
Metric Proposal: Introduced the Representation Locality Score (RLS), a simple metric derived from global inter-layer hidden-state similarity to quantify this phenomenon.
Framework Design: Proposed LoRP, a training-free, one-shot pruning framework that clusters layers by representational similarity and allocates pruning based on the architecture's specific redundancy structure.
Performance Validation: Showed that LoRP consistently improves perplexity and downstream reasoning accuracy across diverse LLM families without requiring post-pruning recovery or fine-tuning.

4. Experimental Results

Experiments were conducted on five open-source LLMs (LLaMA-3.1-8B, OLMo-3-7B, Mistral-Nemo-12B, Qwen3-8B, Qwen3-14B) using nine zero-shot reasoning benchmarks and three perplexity datasets (WikiText-2, C4, PTB).

Perplexity (PPL): LoRP achieved the lowest average perplexity across most models and pruning budgets compared to baselines (ShortGPT, LLM-Streamline, LaCo). The improvement was particularly significant for Qwen models, where contiguous pruning methods failed.
Downstream Accuracy: LoRP preserved zero-shot reasoning performance better than existing methods. For example, on Qwen3-8B with 7/36 layers pruned, LoRP achieved 56.65% average accuracy, outperforming ShortGPT (51.50%) and LaCo (55.82%).
Pruning Patterns: Visualization confirmed that LoRP adapts its pruning pattern to the model: it removes layers from concentrated regions for LLaMA/Mistral (high RLS) and spreads removals across the network for Qwen (low RLS).
Comparison with Iterative Methods: While iterative methods like Shortened LLaMA sometimes achieved lower perplexity (due to adaptation to the calibration objective), LoRP maintained superior downstream generalization accuracy, suggesting better robustness beyond the calibration set.

5. Significance and Claims

The paper claims that representation locality is a critical, architecture-dependent factor for effective depth pruning. The significance of this work lies in shifting the paradigm from fixed, contiguous, or isolated layer scoring to a global, geometry-aware approach.

Modest Claims: The authors explicitly state that their findings suggest redundancy is a structured property of representation geometry. They do not claim LoRP is a universal solution for all compression tasks but rather a framework that improves efficiency by respecting architectural differences.
Limitations: The authors acknowledge that their RLS-guided clustering policy is empirically derived from the evaluated models. They note that the current method relies on a manually specified number of clusters ( $K$ ) based on RLS ranges and does not yet include post-pruning recovery or fine-tuning. Future work is needed to validate generalization across broader architectures, scales, and training recipes, and to develop fully automatic strategies for selecting clustering parameters.

In summary, LoRP demonstrates that by characterizing and adapting to the specific redundancy structure of an LLM, one can achieve more efficient depth compression without sacrificing model performance.