Technical Summary: Adaptive Signal Resuscitation (ASR)

1. Problem Statement

One-shot magnitude pruning, a practical method for compressing vision networks without retraining, often suffers from severe accuracy collapse in high-sparsity regimes (e.g., removing 90% of weights). While existing post-pruning repair methods, such as BatchNorm (BN) recalibration and layer-wise activation scaling (e.g., REFLOW), aim to restore signal, they frequently fail under extreme sparsity.

The core issue identified is a granularity mismatch between the damage caused by pruning and the repair applied. Global magnitude pruning induces heterogeneous channel damage within a single layer: some channels retain informative activation variance, while others collapse to near-zero variance. Existing layer-wise repair methods apply a single scalar correction to the entire layer. When a layer contains a mix of healthy and collapsed channels, the layer-wise correction is driven by the collapsed channels (which have near-zero variance), resulting in an excessively large scaling factor. This factor indiscriminately amplifies both healthy signals and residual noise in collapsed channels, often destabilizing the model and performing worse than simple BN recalibration alone.

2. Methodology: Adaptive Signal Resuscitation (ASR)

ASR is a training-free, post-pruning repair method designed to match the granularity of the repair to the granularity of the damage. It operates before BatchNorm recalibration and requires only forward passes on a small calibration set (64 images).

Core Mechanism

1. Channel-wise Variance Estimation:
   For each output channel $i$ in a convolutional layer, ASR estimates a raw correction factor $\hat{\gamma}_i$ by matching the variance of the pruned activation ($v_{p,i}$) to the variance of the dense activation ($v_{d,i}$) on the calibration set:
   $$\hat{\gamma}_i = \sqrt{\frac{\hat{v}_{d,i}}{\hat{v}_{p,i} + \epsilon}}$$
   where $\epsilon$ is a numerical floor.

2. Data-Driven Shrinkage:
   To prevent the over-amplification of collapsed channels (where $v_{p,i} \approx 0$), ASR introduces a shrinkage step that pulls unreliable estimates toward the identity map ($\gamma_i = 1$).

   • A layer-level baseline $\lambda$ is computed as the median of the post-pruning channel variances within the layer.
   • A reliability weight $s_i$ is calculated based on the ratio of the channel's variance to this baseline:
     $$s_i = \frac{\hat{v}_{p,i}}{\hat{v}_{p,i} + \lambda}$$
   • The final repair factor is a convex combination of the raw estimate and the identity:
     $$\gamma_i = s_i \hat{\gamma}_i + (1 - s_i) \cdot 1$$
   • Behavior: If a channel is collapsed ($v_{p,i} \to 0$), $s_i \to 0$, and $\gamma_i \to 1$ (no correction). If a channel is healthy ($v_{p,i} \gg \lambda$), $s_i \to 1$, and $\gamma_i \approx \hat{\gamma}_i$ (full variance matching).

3. Bias Correction:
   After rescaling, ASR optionally adjusts the convolutional bias to align the first moment (mean) of the repaired activations with the dense activations, addressing mean shifts introduced by pruning.

4. BatchNorm Recalibration:
   The repaired weights and biases are used to compute new running statistics for BatchNorm layers on the calibration set.

3. Key Contributions

• Identification of Failure Mode: The paper demonstrates that layer-wise repair fails in high-sparsity regimes due to heterogeneous channel damage, where a shared scalar is dominated by collapsed channels, leading to over-amplification of noise.
• Shrinkage-Stabilized Repair: ASR introduces a channel-wise repair strategy stabilized by a data-driven shrinkage rule. Unlike naive channel-wise matching, which remains unstable for collapsed channels, ASR suppresses corrections for channels with weak post-pruning signals while preserving corrections for healthier channels.
• Empirical Validation: The method is validated across three datasets (CIFAR-10, CIFAR-100, Imagenette) and four architectures (ResNet-18, ResNet-50, DenseNet-121, VGG-16-BN) under both unstructured and structured (2:4) sparsity.

4. Experimental Results

• High-Sparsity Performance: ASR consistently outperforms layer-wise repair (LW+BN) and often surpasses BN-only recalibration in high-sparsity settings (70%–90%).
  • Example: On ResNet-50 at 90% sparsity (CIFAR-10), ASR achieves 55.6% top-1 accuracy, compared to 41.0% for layer-wise repair and 28.0% for BN-only.
• Granularity Mismatch Visualization: Analysis of per-channel variance shows that layer-wise repair causes systematic over-correction (slope $\alpha \approx 3.84$ in log-log plots), whereas ASR maintains a slope closer to the ideal ($\alpha \approx 1.68$), indicating better alignment with the dense signal.
• Calibration Efficiency: ASR improves early calibration efficiency. The performance gap between ASR and layer-wise repair is most pronounced at small calibration budgets (e.g., 10–20 batches), suggesting ASR stabilizes the activation distribution entering BN layers more effectively.
• Ablation Studies:
  • Channel granularity alone (raw variance matching without shrinkage) is insufficient and performs similarly to or worse than BN-only.
  • The shrinkage mechanism is the critical component that stabilizes the repair.
  • Bias correction provides additional, though secondary, gains.
• Architecture Dependence: The benefits of ASR are most significant in architectures with residual connections (ResNet, DenseNet). In VGG-16-BN (lacking skip connections) at extreme sparsity, BN-only recalibration sometimes remains competitive, as multiplicative repair may introduce instability when channel collapse is too severe to recover.

5. Significance and Claims

The paper claims that ASR addresses a specific failure mode in post-training pruning: the instability of layer-wise repair when channel damage is heterogeneous. By matching the repair granularity to the damage granularity and using shrinkage to suppress unreliable corrections, ASR enables significant accuracy recovery in high-sparsity regimes without retraining.

The authors position ASR as a practical, training-free solution for deploying sparse vision networks under strict memory constraints. They note that the method is specifically designed for convolutional networks with BatchNorm and that extensions to transformer architectures or post-training quantization are left for future work. The paper emphasizes that the gains are largest when pruning induces severe and heterogeneous channel damage, validating the hypothesis that layer-wise assumptions are systematically violated in global magnitude pruning.