Technical Summary: On the Stability of Growth in Structural Plasticity

Problem Statement

Standard deep learning pipelines typically fix the network architecture before training. In contrast, structural plasticity allows for the modification of network architectures during training, primarily through two operators: pruning (removing capacity) and growth (adding capacity). While pruning is well-established and often linked to the Lottery Ticket Hypothesis (LTH), growth is less understood. The paper posits that growth is not simply the inverse of pruning. Pruning selects from units that have participated in training from initialization, whereas growth inserts new units into an already specialized optimization trajectory. The central problem investigated is whether newly inserted units can integrate stably and quickly enough to support adaptive and continual learning, or if the insertion process itself introduces instability that hinders performance.

Methodology

The authors evaluate structural plasticity under matched data streams, optimizers, and compactness targets, comparing three model families:

1. Dense: No sparsification.
2. Prune: Iterative Magnitude Pruning (IMP) applied to masked layers, with surviving weights rewound to an earlier checkpoint (following the LTH protocol).
3. Grow: Starts from a sparse seed and iteratively activates units based on a scoring heuristic (activation frequency or gradient magnitude) until a target compactness is reached. Crucially, Grow does not rewind weights; it studies the late insertion of units into a mature trajectory.

The evaluation employs two distinct metrics to separate the quality of the editing process from the quality of the final architecture:

• Cycle Performance: Accuracy measured during the online structural-editing procedure.
• Winning-Ticket (WT) Performance: Accuracy measured after freezing the discovered mask, reinitializing weights, and retraining from scratch.

Experiments span minimal Multi-Layer Perceptron (MLP) benchmarks (MNIST, Fashion-MNIST) and more complex Convolutional Neural Network (ConvNet) settings (CIFAR-10, CIFAR-100). The study also investigates continual learning scenarios involving sequential class accumulation and repeated distributional shifts.

To isolate the mechanisms of growth failure, the authors introduce several interventions:

• Optimizer State: "Two-Speed" (scaling updates for newborn parameters) and "Moment Transplant" (copying optimizer buffers from incumbent units).
• Insertion: "Net2Wider" (function-preserving widening).
• Selection: "GradMax" (selecting units based on gradient magnitude).
• Trainability: "Rand. Smooth-Leaky" (a randomized, smooth activation function designed to preserve non-zero gradient flow).

Key Contributions and Findings

1. The Asymmetry Between Grow and Prune

The paper demonstrates that the performance gap between growth and pruning is not monolithic; it depends heavily on the complexity of the representation learning regime:

• Small MLPs: In simple settings (e.g., MNIST), Grow and Prune produce similarly retrainable masks with no consistent ordering in final accuracy or trajectory-average accuracy.
• Convolutional Regimes: In harder settings (CIFAR-100), a clear asymmetry emerges. Grow achieves higher final accuracy during the structural-editing cycle (Cycle Performance) but produces masks that, when retrained from scratch, are comparable to or slightly worse than Prune. Conversely, Prune often maintains stronger trajectory-average performance during the edit process and produces subnetworks that retrain robustly.

2. The "Forward-Active, Backward-Starved" Phenomenon

A core mechanistic finding is that newborn units in growth scenarios suffer from a specific integration bottleneck:

• Forward Participation: Newborn units are active immediately upon insertion; they participate in the forward computation and are not "dead" units.
• Backward Disadvantage: Despite being forward-active, newborn units receive significantly weaker gradient signals (backward credit) compared to incumbent units. This "backward-starvation" persists throughout the training cycle, preventing new units from integrating effectively into the optimization trajectory.
• Implication: The growth disadvantage is a process-level issue related to when and how units are inserted, rather than a fundamental inability to discover useful sparse architectures.

3. Interventions and Integration

The study tests various interventions to alleviate the integration bottleneck:

• Optimizer State: "Moment Transplant" (copying optimizer states) shows some promise, while "Two-Speed" (accelerating updates) is less reliable.
• Activation Functions: The Rand. Smooth-Leaky activation function proves most effective. It improves the trainability of newborn units by ensuring better gradient flow, significantly boosting the Cycle Performance of Grow methods, particularly in continual learning settings.
• Selection and Insertion: Changing the selection heuristic (GradMax) or using function-preserving insertion (Net2Wider) does not eliminate the backward-starvation problem.

4. Continual Learning and Time Sensitivity

In continual learning benchmarks stressing plasticity loss (e.g., Permuted MNIST, Random Label tasks):

• Prune remains the most reliable baseline because it preserves mature capacity.
• Grow is competitive only when new units are given sufficient time to integrate before the next distributional shift.
• The combination of Grow + Rand. Smooth-Leaky becomes the most robust growth variant, narrowing the gap to Prune and outperforming it in settings with longer integration windows (e.g., Split-CIFAR100).

Significance and Claims

The paper argues that structural growth should not be evaluated solely as an architecture-search operator (i.e., does it find a good final mask?), but as a time-sensitive optimization process. The success of growth depends critically on insertion stability—the ability of newborn units to integrate quickly into a mature optimization trajectory.

The authors claim that the primary limitation of growth is not the discovery of sparse architectures, but the "birth-time disadvantages" faced by new units:

1. Function Shock: Perturbation of the learned mapping.
2. Cold Start: Lack of accumulated optimizer state.
3. Weak Learning Signal: Disproportionately low gradient credit relative to incumbents.

The paper concludes that improving the trainability and early integration of newborn units (e.g., via activation function design) can make growth a viable alternative to pruning, particularly in adaptive and continual learning systems. However, this requires acknowledging that the "adaptive process" and the "final retrainable architecture" are distinct objects of evaluation, and growth often excels in the former while lagging in the latter unless specific integration mechanisms are employed.