Technical Summary: Beyond Single-Dimensional Compression

Problem Statement

Large Language Models (LLMs) face significant deployment challenges due to high inference costs. While model compression techniques like static parameter pruning and dynamic token-level computation have been developed to reduce memory and compute requirements, they share a common limitation: the existence of an essential sparsity boundary. Beyond a critical sparsity threshold, further aggressive sparsification triggers rapid performance degradation.

Current research typically studies these compression mechanisms in isolation. Static parameter pruning removes structural redundancy before inference, while dynamic token skipping adapts computation per input during inference. The central question this work addresses is whether combining these two distinct mechanisms can delay the onset of performance degradation by distributing the compression burden across multiple dimensions, rather than exhausting the redundancy of a single dimension.

Methodology

The authors propose a minimalist compound sparsity framework designed to independently control parameter sparsity and token-level computation sparsity. The pipeline consists of two sequential stages:

1. Static Parameter Space Compression:

   • The framework applies structured pruning to the pre-trained LLM backbone.
   • It utilizes LoRAP (Low-Rank Approximation Pruning) for attention modules and channel pruning for MLP modules.
   • This reduces the structural cost of the model, converting the full-layer operator $F(\cdot)$ into a sparse backbone operator $F_p(\cdot)$.

2. Dynamic Token-Level Computational Compression:

   • Built upon the pruned backbone, a lightweight, inference-adaptive routing mechanism is introduced.
   • Following SkipGPT, a post-trained router generates binary indicators for each token.
   • Tokens are split into an Active subset (undergoing full forward computation) and an Inactive subset (skipped and passed directly to the next layer).

Definition of Compound Sparsity:
The authors define the global compound sparsity ($S_{comp}$) based on the reduction in effective inference computation:
$$S_{comp} = 1 - (1 - s_p)(1 - s_t)$$
Where $s_p$ is the parameter sparsity ratio and $s_t$ is the token computational sparsity ratio. This formulation allows for flexible allocation of a fixed total sparsity budget between the two dimensions (e.g., balanced allocation where $s_p \approx s_t$).

Key Results

Experiments were conducted using Llama3.1-8B on Redpajama training data, evaluated on language understanding benchmarks (BoolQ, PIQA, HellaSwag, etc.) and language modeling (WikiText-2 PPL).

1. Delayed Degradation:

   • Compound sparsity consistently outperforms single-mechanism compression under the same total sparsity.
   • On understanding tasks, the "essential sparsity" collapse threshold (where performance drops sharply) is delayed from approximately 20% (single-mechanism) to 30% (compound).
   • On language modeling, compound sparsity retains significantly higher performance. At 30% total sparsity, compound sparsity maintains ~70% relative performance, compared to ~60% for token skipping and ~56% for parameter pruning alone.

2. Task Sensitivity:

   • Language modeling tasks are more sensitive to sparsity than multiple-choice understanding tasks. Modeling tasks exhibit early-stage essential sparsity (0–10%) followed by approximate linear degradation, whereas understanding tasks show a more distinct "cliff" in performance.

3. Cross-Dimensional Interference:

   • The study reveals that parameter pruning and token skipping are not entirely independent. A mildly pruned backbone (e.g., 10% parameter sparsity) becomes more sensitive to token skipping.
   • This interference suggests a unified, inherent limit for compound sparsity; while the degradation is delayed, it is not eliminated.

4. Optimal Allocation:

   • Under a fixed total sparsity budget (e.g., 50%), a near-balanced allocation of sparsity between parameters and tokens yields the best performance.
   • Performance follows a convex curve: it improves with mild parameter sparsity but degrades drastically when one dimension is over-compressed.

Key Contributions

1. Perspective Shift: The work investigates essential sparsity from a compound compression perspective, challenging the view that sparsity limits are properties of single mechanisms.
2. Controllable Framework: It introduces a decoupled framework using LoRAP, channel pruning, and per-token dynamic layer skipping, enabling independent manipulation of parameter and token sparsity to analyze trade-offs.
3. Empirical Validation: The study demonstrates that compound sparsity improves the compression-performance trade-off and delays performance decay. However, it also identifies a broader cross-dimensional sparsity boundary that ultimately limits further compression, supporting the principle of "Two Dimensions, One Wall."

Significance and Claims

The paper claims that compound compression provides a practical way to improve LLM compression by distributing the pressure across orthogonal dimensions (parameters and tokens). This approach allows models to achieve higher total sparsity before hitting the performance cliff compared to single-mechanism approaches.

However, the authors maintain a modest stance regarding the ultimate limits of compression. They assert that while sharing the compression burden defers performance degradation, it cannot bypass the inherent sparsity limit. A universal cross-dimensional sparsity wall remains the ultimate constraint. The findings offer principled references for cross-dimensional budget scheduling, suggesting that balanced allocation is the most effective strategy for maximizing model capability under strict resource constraints.

Note: The code for this work is available at https://github.com/EIT-NLP/LLM-Pruning.