Technical Summary: PALS (Percentile-Aware Layerwise Sparsity)

Problem Statement

Large Language Models (LLMs) with billions of parameters impose significant memory and computational burdens during inference. Post-training pruning offers a solution by removing a fraction of weights without retraining. However, standard one-shot pruning methods like Wanda and SparseGPT enforce uniform sparsity, applying the same pruning ratio (e.g., 50%) to every layer of a transformer.

This approach ignores known variations in layer importance. Transformer layers exhibit functional heterogeneity: early layers often build generic representations, middle layers can be redundant, and late layers specialize for output vocabulary. Uniform pruning fails to exploit these differences, potentially removing critical weights from sensitive layers while over-pruning redundant ones, leading to suboptimal perplexity and accuracy.

Methodology: PALS

The authors propose PALS (Percentile-Aware Layerwise Sparsity), a method that dynamically adjusts the pruning ratio for each layer based on activation statistics gathered from a small calibration set. PALS integrates into the Wanda pipeline with negligible overhead and requires no fine-tuning.

Core Algorithm

1. Layer Importance via Activation Percentiles:
   For each layer $l$, activations $A^{(l)}$ are collected over a calibration set. Instead of using the mean activation (which averages out critical signals), PALS computes the 99th percentile of absolute activation magnitudes:
   $$I^{(l)} = \text{Percentile}(|A^{(l)}|, 99)$$
   This metric targets "activation outliers"—features with magnitudes 10–100× larger than the rest—which prior work (Dettmers et al.) has identified as essential for model quality.

2. Normalization and Allocation:
   The raw percentile scores are standardized ($\hat{I}^{(l)}$) to be comparable across layers. The per-layer sparsity target $s^{(l)}$ is then calculated by shifting the global target $s_{\text{target}}$ based on the normalized importance:
   $$s^{(l)} = \text{clip}(s_{\text{target}} + \alpha \cdot \hat{I}^{(l)}, s_{\text{min}}, s_{\text{max}})$$

   • $\alpha$ (Adaptation Strength): Controls how much the sparsity deviates from the target.
   • Conservative Bounds: The method enforces a $\pm 5\%$ band around the target (e.g., for a 50% target, layers are pruned between 45% and 55%). This prevents "information bottlenecks" where heavily pruned layers degrade the entire network's capacity.

3. Pruning Execution:
   Once $s^{(l)}$ is determined for each layer, weights are scored and removed using Wanda's criterion ($|w| \cdot \text{mean}(|a|)$). The bottom $s^{(l)} \times 100\%$ of weights are removed. No weight updates or fine-tuning occur.

Key Finding: Failure of Gradient-Based Allocation

The authors tested a seemingly more principled approach: using gradient magnitudes (or first-order Taylor expansions) to allocate sparsity. This approach failed decisively, producing perplexity scores (47.32) significantly worse than random allocation (24.37) and uniform pruning (12.92). The authors hypothesize that gradients reflect optimization dynamics (local convergence) rather than the discrete impact of removing large fractions of weights.

Experimental Results

Primary Results (LLaMA-2-7B)

On LLaMA-2-7B at 50% unstructured sparsity:

• Uniform Wanda: 12.92 WikiText-2 perplexity.
• PALS: 10.96 WikiText-2 perplexity.
• Significance: This represents a 15.2% improvement in perplexity. The result is statistically significant ($p < 0.001$) across nine independent runs.
• Downstream Tasks: PALS improves zero-shot accuracy on BoolQ, PIQA, HellaSwag, and MMLU compared to Wanda, retaining 93.3% of the dense model's average accuracy.

Architecture Dependence

The benefits of PALS are not universal; they depend on the model's internal layer heterogeneity:

• LLaMA-3-8B: Marginal gain (10.48 vs. 10.45).
• Mistral-7B: No gain (6.31 for both uniform and PALS).
• Interpretation: Newer models (LLaMA-3, Mistral) likely distribute computation more uniformly across layers due to larger training datasets or architectural changes (e.g., Grouped-Query Attention), leaving less room for non-uniform allocation to exploit.

Efficiency

• Memory: Reduces model size by 50% (13.2 GB $\to$ 6.6 GB).
• Throughput: Achieves a 1.41× speedup on NVIDIA A100 GPUs using sparse matrix kernels.
• Overhead: The PALS pipeline adds negligible time to the pruning process, as it reuses activation data already collected by Wanda.

Significance and Claims

The paper makes three primary claims regarding its contribution:

1. Performance Gain via Non-Uniformity: PALS demonstrates that adjusting sparsity ratios per layer based on activation outliers yields significant perplexity reductions in models with heterogeneous layer importance (specifically LLaMA-2), outperforming the standard uniform pruning baseline.
2. Gradient Failure in One-Shot Pruning: The paper documents a striking failure mode where gradient-based importance metrics, successful in smaller networks, produce allocations worse than random for pretrained LLMs. This suggests a fundamental disconnect between gradient magnitude and the impact of discrete weight removal in the one-shot setting.
3. Robustness and Simplicity: The method requires no fine-tuning, adds minimal computational cost, and relies on a simple statistical heuristic (99th percentile) that aligns with the known phenomenon of activation outliers in transformers.

The authors remain modest regarding the universality of the gains, noting that the method's efficacy is contingent on the specific architecture and training dynamics of the target model. If a model's layers are uniformly important, PALS offers no advantage over uniform pruning.