Technical Summary: Automatic Pruning Discovery for Large Language Models

Problem Statement

Large Language Models (LLMs) have achieved remarkable performance but face significant barriers to real-world deployment due to their massive size and computational costs. While existing unstructured pruning methods (e.g., Magnitude, SparseGPT, Wanda) aim to create lightweight models without retraining, they suffer from two critical limitations:

1. Reliance on Manual Expert Knowledge: Current methods depend heavily on human-designed heuristics derived from extensive trial-and-error, resulting in high labor costs and a lack of adaptability.
2. The Outlier Value Issue: The authors identify a previously under-explored phenomenon where high pruning ratios lead to dramatic performance degradation. This is attributed to uniform sparsity, which fails to account for layers with highly skewed weight distributions (outliers). Pruning these sensitive layers disproportionately harms model performance.

Methodology

The paper proposes AutoPrune, a novel paradigm that enables LLMs to automatically design their own optimal pruning algorithms without human expert intervention. The methodology consists of two core components:

1. Graph-driven Chain-of-Thought (GCoT)

To overcome the "black-box" nature of LLMs and the limitations of linear reasoning, the authors introduce GCoT. This module guides the LLM through a structured, multi-stage reasoning process to generate pruning algorithms:

• Analysis: The LLM analyzes model statistics to identify signals relevant to pruning sensitivity.
• Hypothesis: It synthesizes a causal statement linking these signals to parameter importance.
• Conceptual Formula: The hypothesis is formalized into a semi-structured computational relation.
• Computable Concept: The relation is decomposed into modular, machine-readable components.

Unlike standard Chain-of-Thought (CoT), GCoT employs a graph-driven approach rather than a single linear trajectory. It performs temperature-based branching at selected stages to explore multiple reasoning paths. The resulting candidate algorithms are evaluated on a target task (e.g., perplexity on WikiText-2), and the best-performing path is selected to generate the next generation of pruning rules. This process allows the LLM to iteratively refine its own pruning logic based on positive feedback signals.

2. Skew-aware Dynamic Sparsity Allocation (SDSA)

Addressing the outlier value issue, the authors propose SDSA to replace uniform sparsity.

• Skewness Measurement: The method calculates the biased sample skewness of weight magnitudes for each layer. Layers with high positive skewness contain a small fraction of extremely large weights that disproportionately influence output.
• Dynamic Allocation: SDSA uses a normalized skewness score to assign a "protection weight" to each layer. Layers with higher skewness are assigned lower pruning ratios (more preserved), while less sensitive layers are pruned more aggressively.
• Global Calibration: A global sparsity-aware schedule modulates the sensitivity of this allocation based on the overall pruning ratio, ensuring a balanced adaptation trajectory that prevents over-pruning of critical layers.

Key Contributions

1. New Pruning Paradigm: The paper presents the first effort to leverage LLMs to automatically design optimal pruning algorithms for themselves, breaking the dependency on human expert knowledge and manual heuristic design.
2. Identification of the Outlier Value Issue: The authors provide an in-depth analysis revealing that uniform sparsity is the root cause of severe performance degradation under high pruning ratios. They demonstrate that layers with high weight distribution skewness are uniquely sensitive to pruning.
3. GCoT-Driven Self-Pruning: A novel reasoning engine (GCoT) is introduced to structure the LLM's discovery process, enabling the generation of interpretable and high-performing pruning algorithms through multi-path exploration.
4. SDSA Mechanism: A skewness-aware dynamic sparsity allocation strategy is proposed to mitigate the outlier value issue, ensuring stable performance even at aggressive compression levels.

Experimental Results

Extensive experiments were conducted on mainstream LLMs (LLaMA-1 and LLaMA-2 series, ranging from 7B to 70B parameters) across various benchmarks:

• Language Modeling (WikiText): AutoPrune consistently outperformed state-of-the-art training-free baselines (Wanda, SparseGPT, Magnitude) across various sparsity levels (50%, 60%) and model scales. For instance, on LLaMA-1 13B at 60% sparsity, AutoPrune reduced perplexity from 8.56 (Wanda) to 7.63.
• Zero-Shot Tasks: On seven zero-shot tasks (BoolQ, RTE, HellaSwag, etc.), AutoPrune achieved superior or competitive accuracy compared to existing methods across different model sizes.
• Outlier Mitigation: When combined with SDSA, the method significantly reduced perplexity compared to uniform allocation, particularly at high sparsity ratios (e.g., 60-70%), validating the effectiveness of addressing the skewness issue.
• Generalization: The discovered pruning rules transferred effectively to unseen model families (e.g., Pythia-12B) and different sparsity ratios without re-running the search.
• Active Reasoning: The framework was also tested on the "From Passive to Active Reasoning" benchmark, showing that the structured reasoning approach generalizes beyond simple model compression.

Significance and Claims

The paper claims that AutoPrune represents a fundamental shift in how pruning algorithms are designed. By demonstrating that LLMs can autonomously discover effective pruning strategies, the work challenges the necessity of manual expert intervention in algorithm design. Furthermore, by identifying and solving the outlier value issue through SDSA, the paper provides a more robust solution for high-ratio pruning, enabling the deployment of highly compressed LLMs without the severe performance drops previously observed. The authors position this work as a step toward automated, adaptive sparsity design that is both interpretable (via GCoT) and effective.