Technical Summary: Ghosted Layers – Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs

1. Problem Statement

Layer pruning is a practical compression technique for Large Language Models (LLMs) that removes entire Transformer decoder blocks to reduce model size and inference cost without requiring custom kernels or architectural changes. However, removing these blocks creates a boundary activation mismatch. The hidden state passed to the next surviving layer differs significantly from the distribution the layer was originally trained to process. This distribution shift propagates through downstream layers, leading to substantial performance degradation in both perplexity and downstream task accuracy.

Existing training-free recovery methods attempt to mitigate this mismatch but suffer from structural limitations:

• ReplaceMe approximates the pruned block's computation via a linear transformation absorbed into surviving weights but does not directly address the boundary mismatch.
• LinearPatch directly targets the boundary activation mismatch by inserting a linear operator parameterized as a Hadamard-rotated channel-wise scaling. Crucially, this parameterization is symmetric by construction, restricting the search space to a strict subspace of linear operators and preventing the method from reaching the global optimum of the alignment objective.

Empirical analysis shows that these methods leave a significant portion of the boundary activation error uncorrected, suggesting that the constrained nature of their operators is insufficient to fully recover the pruned model's performance.

2. Methodology: Ghosted Layers

The authors propose Ghosted Layers, a training-free, plug-and-play recovery module that addresses the boundary activation mismatch by solving for the unconstrained optimal linear operator.

Core Formulation

The method formulates recovery as an activation alignment problem. Let $X_{pre}$ be the hidden state input to the first pruned layer and $X_{post}$ be the hidden state output of the last pruned layer (input to the next surviving layer) in the original, unpruned model. The goal is to find a linear operator $W$ such that $X_{pre}W \approx X_{post}$.

The authors define the boundary activation gap as $\Delta = X_{post} - X_{pre}$. They reparameterize the operator as $W = I + M$, transforming the objective into minimizing the reconstruction error of the gap:
$$\min_{M \in \mathbb{R}^{C \times C}} \| X_{pre}M - \Delta \|_F^2$$

Closed-Form Solution

Unlike previous methods that optimize over constrained subspaces, Ghosted Layers derives a closed-form minimum-norm least-squares solution over the full space of linear operators $\mathbb{R}^{C \times C}$:
$$M^* = X_{pre}^\dagger \Delta$$
where $X_{pre}^\dagger$ is the Moore-Penrose pseudoinverse of the pre-boundary activations. The final operator is $W^* = I + M^*$.

Implementation

1. Calibration: A small set of calibration sequences (e.g., 128 sequences from C4) is passed through the original unpruned model to collect $X_{pre}$ and $X_{post}$ via forward pre-hooks.
2. Computation: The optimal operator $M^*$ is computed offline using a regularized least-squares solver (or SVD) to ensure numerical stability.
3. Deployment: The operator $W^*$ is inserted at the pruning boundary via a forward hook. The operation is $x_{new} = x W^* = x + x M^*$, where the additive term $x M^*$ estimates the incremental update the pruned layers would have contributed.

Theoretical Advantage

The paper proves that the optimal operator $W^*$ contains a substantial anti-symmetric component. Since LinearPatch restricts its search to symmetric matrices (where $W = W^\top$), it structurally cannot access this anti-symmetric component. Ghosted Layers, by operating over the unconstrained space, captures this component, theoretically achieving the true optimum of the alignment objective.

3. Key Contributions

1. Formulation of Post-Pruning Recovery: The authors formulate recovery as an activation alignment problem and derive its closed-form optimal solution directly from boundary activations.
2. Unconstrained Optimality: They demonstrate that this formulation yields an unconstrained optimum that includes a significant anti-symmetric component, which is inaccessible to symmetric constructions like LinearPatch.
3. Ghosted Layers Module: They propose a training-free, plug-and-play module compatible with any pruning criterion and model architecture. It consistently outperforms prior training-free baselines at matched inference costs.

4. Experimental Results

The method was evaluated across multiple LLM backbones (LLaMA-3-8B, LLaMA-3.1-8B, DeepSeek-R1-Distill-LLaMA-8B, and others in the appendix) and pruning strategies (LLM-Streamline, ShortGPT, Shortened LLaMA).

• Accuracy: Ghosted Layers consistently achieves the highest or competitive average zero-shot accuracy on nine commonsense reasoning benchmarks (ARC, HellaSwag, WinoGrande, etc.) compared to training-free baselines. For example, on LLaMA-3.1-8B with 7 layers pruned, Ghosted Layers achieved 60.01% average accuracy, outperforming LinearPatch (Rotate) at 53.27% and ReplaceMe (LS) at 59.68%.
• Perplexity: The method significantly reduces perplexity on WikiText-2, C4, and Penn Treebank. In aggressive pruning scenarios (11 layers removed), the margin of improvement over baselines widens, with Ghosted Layers achieving the lowest perplexity in most combinations.
• Efficiency: Despite solving for a full $C \times C$ matrix, Ghosted Layers incurs no additional inference cost compared to LinearPatch. Both methods fuse their operations into a single dense $C \times C$ matrix multiplication at the boundary. Empirical latency measurements confirm they are indistinguishable within noise margins.
• Calibration Sensitivity: The method is robust to the size of the calibration set; accuracy saturates with as few as 32 sequences, and the choice of calibration dataset (C4 vs. WikiText-2) has minimal impact on performance.
• Fine-Tuning: Even when combined with lightweight knowledge distillation, Ghosted Layers maintains a performance edge over fine-tuned LinearPatch variants, suggesting the closed-form initialization provides a superior starting point.

5. Significance and Claims

The paper claims that solving boundary activation alignment at its unconstrained optimum is sufficient to substantially recover the performance of layer-pruned LLMs without retraining.

The significance of this work lies in demonstrating that the performance degradation in pruned models is not merely a result of missing capacity but of a correctable distribution shift. By moving beyond constrained operator subspaces (like symmetric matrices) to the full space of linear maps, Ghosted Layers recovers structural information (specifically anti-symmetric components) that previous methods missed. This approach offers a simple, computationally efficient, and highly effective solution for deploying smaller, pruned LLMs while preserving their original capabilities.