Growing Networks with Autonomous Pruning

1. Problem Statement

Convolutional Neural Networks (CNNs) have achieved state-of-the-art performance in image classification, but they often suffer from excessive parameter counts, leading to high computational costs and memory usage. Current approaches to address this include:

• Neural Architecture Search (NAS): Automates design but remains computationally expensive as it requires training many models with different hyperparameters.
• Pruning: Starts with a large, over-parameterized network and removes weights. However, this requires training a massive model first (risking overfitting) and relies on manual initial architecture design.
• Growing Networks: Start small and add neurons. However, these methods often struggle to prevent redundancy, potentially resulting in networks larger than necessary.

The core problem addressed by this paper is the lack of a unified method that can dynamically adjust network topology and size during training to find the optimal balance between high classification accuracy and extreme sparsity, without relying on a pre-defined, over-parameterized initial architecture.

2. Methodology: Growing Networks with Autonomous Pruning (GNAP)

GNAP is a novel framework that combines growth and autonomous pruning in an alternating optimization process. The network starts with a small architecture and evolves during training.

A. Architecture Design

• Dense Connectivity: Inspired by DenseNets, GNAP connects every layer to all subsequent layers within "dense blocks."
• Dynamic Structure: Unlike fixed architectures, the number of layers per block and the number of channels (neurons) per layer change dynamically.
• Blocks: The network consists of $B$ dense blocks separated by pooling layers. The number of blocks is fixed, but the internal width and depth vary.

B. Autonomous Pruning (The "Pruning" Mechanism)

GNAP utilizes a differentiable pruning strategy based on Gradient Descent:

• Gating Mechanism: Every weight (or convolutional kernel) is associated with a learnable "gate" parameter.
• Gumbel-Softmax Sampling: During the forward pass, gates are sampled from a Gumbel-Softmax distribution to produce binary values (0 or 1). If a gate is 0, the weight is effectively pruned.
• Loss Function: The total loss $L$ combines the prediction loss ($L_{pred}$, e.g., cross-entropy) and a sparsity loss ($L_{sparsity}$):
  $$L = L_{pred} + \alpha \cdot ||g||_1$$
  Where $g$ represents the gating variables and $\alpha$ is a weighting factor. The model is incentivized to minimize the prediction loss while driving the number of open gates toward zero.
• Flexibility: This mechanism supports both unstructured pruning (individual weights) and structured pruning (entire convolutional kernels), depending on the application needs.

C. Growing Phases (The "Growth" Mechanism)

When the network converges to a "saturation point" (where adding more training epochs yields diminishing returns on accuracy or sparsity), a growing phase is triggered.

• Trigger Condition: Growth occurs when the relative change in the number of open gates falls below a threshold $\theta$, indicating the topology has stabilized.
• Where to Add: Neurons are added everywhere: $C$ new neurons are added to every existing layer, and a new layer is added to the end of every block. All new neurons connect to all preceding neurons.
• Weight Initialization (Key Innovation):
  • New-to-Old Connections: To prevent "garbage" data from destabilizing trained neurons, new weights connecting new neurons to old ones are initialized using a specific strategy. The authors propose initializing these weights to be orthogonal to the existing weight space using Singular Value Decomposition (SVD). This ensures new neurons explore new directions in the feature space rather than duplicating existing ones.
  • Old-to-New Connections: Weights connecting old neurons to new ones are initialized from a Gaussian distribution.

D. Training Procedure

1. Training: The network trains for $E$ epochs using Adam optimizer.
2. Alternating: The process alternates between training (pruning via gradient descent) and growing phases.
3. Early Stopping: A modified early stopping criterion saves models that offer the best trade-off between accuracy and the number of active links (parameters).
4. Refinement: After the final growing phase, a refinement phase occurs with a higher sparsity weight ($\alpha$) to further prune the network.

3. Key Contributions

1. Dynamic Topology Optimization: A new approach that adapts network size and topology dynamically during training via an alternating growing and pruning strategy, eliminating the need for a fixed, over-parameterized starting point.
2. Orthogonal Weight Initialization: A novel initialization strategy for new neurons that uses SVD to ensure they are orthogonal to existing weights, preventing redundancy and improving convergence.
3. Unified Pruning Framework: The ability to perform both structured (kernel-level) and unstructured (weight-level) pruning within the same differentiable framework.
4. Extreme Sparsity with High Accuracy: Demonstration that extremely sparse networks can be trained from scratch without sacrificing classification performance.

4. Experimental Results

The authors evaluated GNAP on MNIST, CIFAR-10, and CIFAR-100.

• MNIST: Achieved 99.44% accuracy with only 6,234 parameters (using structured pruning). This is comparable to or better than state-of-the-art (SOTA) methods while using significantly fewer parameters.
• CIFAR-10: Achieved 92.22% accuracy with 157,765 parameters (unstructured pruning). This outperforms many SOTA sparse methods (like SNIP, GraSP, LTH) that rely on pre-trained dense networks (e.g., ResNet-32, VGG-19) and typically require more parameters to achieve similar accuracy.
• CIFAR-100: Achieved 66.56% accuracy with 179,857 parameters.
• Comparison: GNAP consistently produced models with fewer parameters than SOTA methods (e.g., SimpleNet, MCUNet, RigL) while maintaining competitive or superior accuracy. Notably, GNAP does not require a pre-defined dense base architecture.
• Analysis:
  • Growth vs. Pruning: The growing phases allowed the model to explore better solutions, paradoxically leading to smaller final models than if growth had not occurred.
  • Deterministic Convergence: The gating mechanism converged to a state where gates were almost strictly 0 or 1, effectively removing the stochasticity of the Gumbel-Softmax during inference.
  • Redundancy Removal: The pruning process successfully eliminated over 500,000 weights that initially had high activation probabilities, proving the efficacy of the autonomous pruning.

5. Significance

This paper presents a significant shift in how neural network architectures are designed and trained.

• Efficiency: It challenges the paradigm of "train large, then prune" by demonstrating that one can "grow small, then optimize," saving significant computational resources.
• Automation: It reduces the manual burden of architecture design (hyperparameter tuning, layer count selection) by letting the network self-determine its optimal size.
• Scalability: The method is applicable to both structured and unstructured pruning, making it versatile for various hardware constraints (e.g., sparse matrix multiplication vs. kernel removal).
• Future Potential: The authors suggest this approach is highly promising for continual learning and non-IID data scenarios, where static architectures often fail.

In summary, GNAP offers a robust, automated, and highly efficient framework for creating sparse neural networks that rival the performance of much larger, manually designed models.