Technical Summary: LLMs can Compress LLMs: Adaptive Pruning by Agents

Problem Statement

As Large Language Models (LLMs) scale, their deployment is hindered by substantial computational and memory requirements. While post-training pruning offers a solution to reduce these costs without expensive retraining, existing methods face critical limitations:

1. Uniform Heuristics: Current approaches like SparseGPT and Wanda rely on uniform sparsity ratios or hand-crafted heuristics to determine per-layer pruning. This "one-size-fits-all" strategy fails to account for the heterogeneous importance of different layers in preserving model capabilities.
2. Factual Knowledge Degradation: Recent benchmarks (e.g., LLM-KICK) reveal that pruned LLMs, particularly those using structured pruning, suffer from catastrophic degradation in factual knowledge. Structured methods can lose over 97% of accuracy on knowledge-intensive tasks (e.g., FreebaseQA) even at modest sparsity ratios (25-30%), despite negligible increases in perplexity. This indicates that perplexity is an inadequate metric for evaluating the true capability of compressed models.

Methodology

The authors propose Agent-Guided Pruning, a framework where a foundation model acts as an adaptive pruning agent to intelligently select which layers to prune at each iteration. The method operates without retraining and is model-agnostic.

Core Components

1. Layer-Wise Sensitivity Profiling:

   • The system constructs sensitivity profiles for each linear layer by combining Wanda-inspired weight-activation metrics (magnitude of weights multiplied by input activation norms) and gradient importance scores (inspired by Wanda++).
   • These metrics are normalized using z-score standardization to allow for model-agnostic comparison across heterogeneous layer types. Negative z-scores indicate layers with below-average sensitivity (safer to prune), while positive z-scores indicate higher sensitivity.

2. LLM Agent Decision Making:

   • A foundation model (specifically gemini-3-flash-preview) acts as the agent. At each iteration, it receives:
     • Current and target global sparsity.
     • Sorted layer profiles (z-scores).
     • Current and baseline perplexity.
     • Feedback summaries from previous iterations.
   • The agent reasons about which layers to prune to balance sparsity gain with model health, outputting structured JSON decisions regarding specific layer sparsity increments.

3. Self-Reflection Mechanism:

   • The agent operates in an iterative feedback loop. After each pruning step, it receives a summary of the previous decision's outcome, including sparsity gain, perplexity change, and a qualitative assessment.
   • This allows the agent to learn from past decisions, recognizing patterns (e.g., prioritizing layers with highly negative z-scores) and adjusting its strategy (e.g., becoming more conservative as perplexity degrades).

4. Checkpoint Rollback:

   • A safety mechanism monitors perplexity degradation. If the perplexity increase exceeds a threshold (15%), the system reverts to the previous checkpoint, discards the current iteration's decisions, and provides negative feedback to the agent to encourage a more conservative approach.

Key Contributions

The paper outlines five primary contributions:

• First LLM-as-Agent Framework: Introduces the first framework using an LLM as an adaptive pruning agent for iterative layer selection, eliminating the need for manual heuristic design.
• Self-Reflection Mechanism: Develops a mechanism enabling the pruning agent to learn from previous decisions and refine its strategy through iterative feedback.
• Model-Agnostic Profiling: Proposes a statistical profiling approach using z-score normalization of Wanda metrics and gradient importance to compare heterogeneous layer types.
• Checkpoint Rollback: Implements a safety net that maintains model quality by reverting unsuccessful attempts, demonstrating effective self-correction with low rollback rates.
• Empirical Superiority: Achieves substantial improvements over structured pruning baselines, specifically addressing the factual knowledge degradation problem.

Experimental Results

The approach was evaluated on Qwen3-4B and Qwen3-8B models, targeting approximately 45-50% sparsity. Results were compared against structured pruning baselines (2:4 and 4:8 patterns).

• MMLU Accuracy: The agent-guided method achieved a 56% relative improvement in MMLU accuracy over the best-performing structured baseline (4:8) on the 8B model.
• Factual Knowledge Retention: On the FreebaseQA benchmark, the method demonstrated a 19× improvement in accuracy retention compared to the 4:8 structured baseline (25.16% vs. 1.33% for Qwen3-8B), directly addressing the catastrophic collapse observed in prior work.
• Perplexity: The method resulted in 69% lower perplexity degradation compared to the 4:8 baseline.
• Efficiency and Stability: The framework required no retraining. Across 21–40 iterations, the agent exhibited effective self-correction with low rollback rates (9.5% for Qwen3-8B and 10% for Qwen3-4B), indicating that the agent successfully learned to navigate the trade-off between sparsity and model quality.

Significance and Claims

The paper claims to establish a new paradigm where foundation models can effectively guide the compression of other foundation models. The significance of this work lies in:

1. Solving the Knowledge Degradation Problem: It directly addresses the disconnect between perplexity and task performance, showing that adaptive, agent-guided pruning can preserve critical factual knowledge pathways that uniform or structured pruning destroys.
2. Meta-Learning in Compression: It demonstrates that pruning policies can be refined through experience (meta-learning) rather than relying on fixed importance metrics, adapting to the heterogeneous sensitivities of modern LLM layers.
3. Automation: It opens possibilities for fully automated neural architecture optimization without manual heuristic design or expensive retraining.

The authors conclude that their results establish the viability of using LLM agents to compress LLMs in ways that preserve capabilities missed by traditional metrics and static heuristics.