Rethinking Layer Redundancy in Large Language Models: Calibration Objectives and Search for Depth Pruning

1. Problem Statement

Large Language Models (LLMs) incur significant deployment costs due to their scale. Depth pruning, which removes entire Transformer blocks, is a structured compression technique that reduces inference costs linearly with the number of retained layers.

Prior research in depth pruning has largely operated under a "structural view" of redundancy. This perspective assumes that layer importance is an intrinsic property of the pretrained network, independent of the downstream task. Consequently, previous works focus on finding a single "universal" ranking of layers (via importance criteria like cosine similarity or magnitude) and applying sophisticated search algorithms to identify the optimal subset of layers to remove.

The Core Question: Is layer redundancy truly a structural property of the model, or is it a functional property jointly determined by the model and the specific evaluation objective (calibration loss)? The authors hypothesize that a layer deemed redundant for language modeling (perplexity) might be critical for downstream reasoning, suggesting that a universal ranking is insufficient.

2. Methodology

To empirically test the structural vs. functional views, the authors designed a controlled experiment that disentangles two key axes of depth pruning: the Calibration Objective and the Search Algorithm.

• Problem Formulation: Depth pruning is framed as a subset selection problem:
  $$S^* = \arg \min_{|S|=k} \mathcal{L}(D; f_{\theta_0} \odot m_S)$$
  Where $S$ is the set of removed layers, $\mathcal{L}$ is the loss function, and $D$ is the calibration dataset.
• Experimental Variables:
  • Models: Three open-weight LLM families (LLaMA3 8B, LLaMA3.1 8B, Qwen3 8B) with varying training pipelines but comparable scales.
  • Calibration Objectives ($\mathcal{L}$):
    1. Perplexity (Task-Agnostic): Negative log-likelihood on the C4 dataset.
    2. Task Likelihood Margin (Task-Centric): The log-likelihood gap between correct and incorrect answers on Commonsense 170k.
  • Search Algorithms: Seven distinct algorithms were tested under each objective to isolate the impact of the search strategy:
    1. One-shot top-k selection
    2. Greedy iterative deletion
    3. Beam search
    4. Prior-guided Genetic Algorithm (GA)
    5. Prior-guided Bayesian Optimization (BO)
    6. Constrained Binary Optimization (CBO)
    7. Fast-block-select
• Evaluation:
  • Models pruned via Perplexity were evaluated on WikiText-2, C4, and LAMBADA.
  • Models pruned via Task Likelihood Margin were evaluated zero-shot on commonsense reasoning benchmarks (HellaSwag, WinoGrande, ARC, PIQA, BoolQ).
  • Pruning Budgets: Two levels of compression ($M=7$ and $M=9$ layers removed).

3. Key Contributions

1. Functional View of Redundancy: The paper articulates and empirically validates that layer redundancy is not an intrinsic structural property but a functional one dependent on the evaluation objective.
2. Disentanglement of Search and Objective: By holding one variable fixed while varying the other across three model families, the authors isolate the source of performance gains, addressing the "metric-search confound" common in prior literature.
3. Systematic Misalignment: The authors identify a systematic disconnect between calibration perplexity and downstream reasoning accuracy, demonstrating that optimizing for one does not guarantee optimization for the other.

4. Key Results

A. Objectives Reshape Pruning Patterns (Section 5.1)

• Perplexity Calibration: Pruning patterns were highly consistent across different search algorithms. Removals concentrated in contiguous mid-to-late layer clusters (e.g., layers 9–12 and 22–26 in LLaMA3).
• Task Likelihood Calibration: Pruning patterns were scattered across the network and varied significantly depending on the search algorithm used.
• Conclusion: The two objectives produce structurally different sets of "redundant" layers, refuting the idea of a single universal redundancy map.

B. Perplexity vs. Accuracy Misalignment (Section 5.2)

• The authors compared the ranking of pruned models based on calibration perplexity against their downstream reasoning accuracy.
• Result: Spearman correlations ranged from $\rho = -0.78$ to near zero. In most conditions, the correlation was negative.
• Implication: The model configuration that minimizes perplexity is often not the one that maximizes downstream reasoning accuracy.

C. Search Algorithms are Secondary (Section 5.3)

• Within-Objective Variance: When the calibration objective was fixed, different search algorithms converged to solutions with very similar performance.
  • For task-likelihood pruning, accuracy differences between the best and worst search algorithms were typically 1–3 percentage points.
  • Even complex methods (like BO or GA) offered diminishing returns compared to simpler methods (like One-shot or Greedy) once the objective was fixed.
• Cost-Benefit: Simple methods (One-shot, Fast-block-select) achieved near-optimal results in minutes, while complex methods took hours for marginal gains.
• Conclusion: The choice of calibration objective has a far greater influence on the final model performance and structure than the choice of search algorithm.

5. Significance and Implications

• Paradigm Shift: The paper challenges the prevailing assumption that layer importance is static. It suggests that researchers should stop searching for a "universal" layer ranking and instead focus on objective-conditioned redundancy.
• Practical Efficiency: For practitioners, the results imply that investing computational resources into complex search algorithms (e.g., evolutionary search, Bayesian optimization) yields limited benefits if the calibration objective is suboptimal.
• Design Priority: The most critical design decision in depth pruning is the calibration objective. Selecting an objective that aligns with the intended downstream task (e.g., using a task-specific margin loss for reasoning tasks rather than generic perplexity) is more impactful than refining the search strategy.
• Future Directions: Future work should prioritize the design of calibration objectives and the study of how different tasks define redundancy, rather than solely focusing on improving search heuristics.

In summary, the paper demonstrates that in LLM depth pruning, the objective defines the landscape, while the search algorithm merely navigates it. A poor objective cannot be fixed by a sophisticated search, but a well-chosen objective allows even simple search methods to find high-quality solutions.