CRePE: Convolution-aware Relative Importance in Post-training Pruning with Efficient Search

Technical Summary: CRePE and PHO for Post-Training Pruning

Problem Statement
Deploying Large Language Models (LLMs) incurs significant memory and computational costs. Post-training pruning (PTP) offers a training-free solution by removing weights without retraining. While recent methods like RIA (Relative Importance) have achieved state-of-the-art accuracy by normalizing weights via row and column sums, they suffer from two fundamental limitations:

1. 1D Limitation: RIA relies solely on 1D cross-shaped (row/column) directional information, ignoring the inherent 2D structure of weight matrices where diagonal neighbors contribute to importance.
2. Uniform Weighting: RIA treats row and column contributions equally. Preliminary experiments indicate these directions contribute asymmetrically to pruning quality (e.g., row-wise RI yields lower perplexity than column-wise RI on LLaMA-2-7B), making uniform treatment suboptimal.

Additionally, optimizing the hyperparameters for improved scoring methods typically requires extensive search time. A naive PPL-based hill-climbing approach for finding optimal coefficients takes approximately 11 hours, undermining the efficiency advantage of PTP.

Methodology
The paper proposes CRePE (Convolution-aware Relative Importance in Post-training Pruning with Efficient Search) and PHO (Proxy-based Hyperparameter Optimization).

1. CRePE: Enhanced Importance Scoring
CRePE extends the RIA scoring mechanism in two ways:

• 2D Neighborhood Context: It introduces a neighborhood term $N^{(k)}_{ij}$ that captures local 2D context around each weight $W_{ij}$, excluding the cross-shaped neighbors already covered by RIA. This term is computed efficiently via 2D convolution of the absolute weight matrix with a masked kernel.
• Adaptive Coefficients: It assigns distinct, learnable coefficients ($\alpha, \beta, \gamma$) to the row, column, and neighborhood terms, respectively. This allows the model to explicitly account for the asymmetric contributions of these components.
• Formula: The final importance score is calculated as:
  $$S^{CRePE}_{ij} = \left( \alpha \cdot \frac{|W_{ij}|}{\sum_k |W_{ik}|} + \beta \cdot \frac{|W_{ij}|}{\sum_k |W_{kj}|} + \gamma \cdot \frac{|W_{ij}|}{N^{(k)}_{ij}} \right) \cdot \|X_j\|_2$$
  When $\alpha=\beta=1$ and $\gamma=0$, CRePE reduces to RIA.

2. PHO: Efficient Hyperparameter Search
To avoid the prohibitive cost of PPL-based search for the hyperparameters ($k, \alpha, \beta, \gamma$), the authors propose PHO:

• Proxy Discovery: Through Spearman correlation analysis across diverse models, the authors found that the Gini coefficient of the importance scores of pruned weights in the first-layer $q\_proj$ module ($Gini_{pruned}$) exhibits a strong positive correlation with PPL ($\rho = 0.953$).
• Search Framework: PHO leverages $Gini_{pruned}$ as a surrogate objective. It employs a two-stage process:
  1. Proxy Screening: A CMA-ES (Covariance Matrix Adaptation Evolution Strategy) sampler generates 60 candidates, evaluating them solely on $Gini_{pruned}$ (approx. 6 seconds per candidate).
  2. PPL Verification: Full model pruning and PPL evaluation are performed only on the top 10 candidates (approx. 90 seconds each).
• Efficiency: This reduces the total search time from ~11 hours to ~20 minutes (a 30× speedup).

Key Contributions

• CRePE: A new scoring method that integrates 2D neighborhood context and adaptive coefficients to model the differential importance of row, column, and neighborhood terms.
• PHO: An efficient optimization framework that uses the Gini coefficient of pruned weights as a proxy for PPL, drastically reducing hyperparameter search time.
• Generalization: Demonstrated consistent improvements over baselines (SparseGPT, Wanda, RIA) across diverse models (LLaMA-1/2/3, Qwen-2.5/3/3.5, Phi-4, DeepSeek) and sparsity settings (unstructured, 2:4, 4:8).
• Orthogonality: Verified that CRePE can be combined with existing techniques, including Channel Permutation, non-uniform sparsity allocation (OWL, AlphaPruning, DLP), and re-pruning methods (DSnoT).

Experimental Results

• Perplexity: CRePE consistently achieves lower perplexity than RIA. For example, on LLaMA-2-7B with 50% unstructured pruning, CRePE achieves a PPL of 6.74 compared to RIA's 6.81. Under 2:4 structured pruning, the gap widens (10.74 vs. 11.29), suggesting refined scoring is particularly beneficial under stricter structural constraints.
• Zero-Shot Performance: On LLaMA-2-7B, CRePE achieves the highest average zero-shot accuracy (62.18%) across six tasks, outperforming RIA (61.14%).
• Hyperparameter Transferability: Optimal hyperparameters found by PHO on LLaMA-2-7B transfer effectively to LLaMA-2-13B without re-tuning, maintaining performance gains across model scales.
• Combinations: When combined with Channel Permutation (LSA CP) or non-uniform allocation (DLP), CRePE yields further reductions in perplexity, confirming its orthogonality to these techniques.

Significance and Limitations
The paper claims that CRePE provides a practical, training-free method to significantly improve LLM pruning accuracy while maintaining the efficiency of scoring-only approaches. The introduction of PHO ensures that the necessary hyperparameter tuning does not negate the time-saving benefits of PTP.

The authors note several limitations:

• At extreme sparsity levels (>70%), the performance gap between CRePE and RIA narrows, suggesting that importance scoring alone may be insufficient and weight reconstruction might be required.
• The strong correlation between $Gini_{pruned}$ and PPL is empirical; a rigorous theoretical analysis remains an open question.
• The method is currently limited to dense linear layers and has not been extended to structured components like Mixture-of-Experts (MoE) routing.
• The impact of the additional convolution operation across diverse hardware environments has not been systematically evaluated.