🤖 machine learning

Finding Sparse Subnetworks in One Training Cycle via Progressive Magnitude-Based Pruning

This paper proposes and validates a single-training-cycle method called progressive magnitude-based pruning, which gradually increases sparsity during training and demonstrates superior accuracy at high sparsity levels compared to iterative and initialization-based baselines like the Lottery Ticket Hypothesis, SNIP, and GraSP.

Original authors: Romana Qureshi, Hafida Benhidour, Said Kerrache, Nahlah Aljeraisy

Published 2026-06-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Romana Qureshi, Hafida Benhidour, Said Kerrache, Nahlah Aljeraisy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, over-enthusiastic student who is trying to learn a subject. This student has a brain full of billions of connections, but many of them are just "noise"—they aren't actually helping the student solve the problem. In fact, having too many connections makes the student slow, clumsy, and hard to carry around (like trying to fit a giant library into a backpack).

This paper is about a new, efficient way to teach this student how to be smart but small, all in one single school year, rather than making them repeat the whole year over and over again.

Here is the breakdown of their approach using simple analogies:

The Problem: The "Lottery Ticket" is Too Expensive

Scientists previously discovered something called the "Lottery Ticket Hypothesis." They found that inside a giant, messy neural network (the student's brain), there is a tiny, perfect "winning ticket" (a small, efficient sub-network) that could solve the problem just as well as the giant one.

However, finding this winning ticket was like trying to find a needle in a haystack by building a new haystack, checking it, throwing it away, and building a new one. The old method required:

Train the giant network.
Cut out the weak parts.
Reset the remaining parts back to how they were at the very beginning.
Start over and train again.
Repeat this cycle many times.

This took a huge amount of time and computer power, defeating the purpose of trying to make the model smaller and faster.

The Solution: The "Progressive Gardener"

The authors propose a new method called Progressive Magnitude-Based Pruning. Instead of resetting the garden and starting over, they act like a gardener who trims a plant while it is growing.

Here is how their "one-cycle" method works:

The Linear Schedule (The Slow Trim): Imagine the student is in a 200-day course. Instead of cutting 50% of the connections on day 1, the authors start cutting a tiny bit every single day. By the end of the course, they have gently removed 90% of the connections. This gives the network time to adjust and learn how to function with fewer connections, rather than being shocked by a sudden massive cut.
The Magnitude Rule (Cutting the Weakest): How do they decide what to cut? They look at the "strength" (magnitude) of every connection. If a connection is weak (close to zero), it's like a twig that isn't holding much weight. They cut the weakest twigs first.
No Regrowth (The One-Way Door): Once a connection is cut, it stays cut. They don't let it grow back. This keeps the process simple and ensures the network gets smaller and smaller, never getting bigger again.
The "Active" Check: They only look at the connections that are still alive to decide what to cut next. They ignore the ones that are already dead (zeroed out). This ensures they are always cutting the weakest remaining links.

The Results: Small but Mighty

The authors tested this "Progressive Gardener" on standard tests (like recognizing handwritten numbers or small images) and compared it to the old "reset-and-retrain" methods.

Speed: They did it in one single training cycle. No resetting, no restarting.
Performance: Surprisingly, their "one-shot" method was often better than the old methods that took many cycles.
- On a standard test (CIFAR-10), their method got 95.12% accuracy with a very sparse network, while the old "Lottery Ticket" method only got 90.5% with similar sparsity.
- Even when they cut away almost everything (leaving only 2% of the connections), their method still performed better than the competition.

The "Sweet Spot"

The paper also analyzed how much they could cut before the student started failing. They found a "sweet spot" between 70% and 85% sparsity (meaning 70-85% of the connections are gone).

In this range, the student's performance barely dropped at all (less than 0.1% difference from the full, giant network).
It's like removing 8 out of 10 books from a library, but the student can still answer every question just as well as before.

The Bottom Line

This paper claims that you don't need to go through the exhausting process of "train, cut, reset, retrain" to find a small, efficient neural network. Instead, you can just gradually trim the weak parts while the network learns, and you will end up with a tiny, fast, and highly accurate model in half the time (or less).

It's a simpler, faster way to shrink AI models without losing their smarts.

Technical Summary: Finding Sparse Subnetworks in One Training Cycle via Progressive Magnitude-Based Pruning

1. Problem Statement

Deep neural networks have achieved remarkable success in computer vision and natural language processing but face significant deployment challenges in resource-constrained environments (mobile, edge, IoT) due to their computational and memory demands. Neural network pruning offers a solution by removing less important parameters to reduce model size while preserving predictive performance.

However, existing paradigms suffer from specific limitations:

Iterative Pruning (e.g., Lottery Ticket Hypothesis - LTH): While LTH demonstrates that sparse "winning tickets" exist within dense networks, finding them requires multiple complete training cycles (train, prune, reset weights to initialization, retrain). This introduces substantial computational overhead that undermines the efficiency goals of pruning.
Initialization-Based Pruning (e.g., SNIP, GraSP): These methods prune before training begins based on connection sensitivity or gradient flow. While computationally cheaper, they commit to a fixed sparse structure before weight importance can evolve during learning.
Dynamic Sparse Training: Methods that allow connectivity to change via pruning and regrowth introduce implementation complexity and require careful tuning of regrowth schedules.

The central problem addressed is the lack of an efficient, single-cycle pruning method that can identify high-performing sparse subnetworks without the computational cost of iterative retraining or the rigidity of pre-training pruning.

2. Methodology

The authors propose Progressive Magnitude-Based Pruning, a method that identifies sparse subnetworks during a single training cycle by gradually increasing sparsity.

Core Mechanism

Linear Sparsity Schedule: The method employs a linear schedule to increase sparsity from an initial value ( $s_i$ ) to a target value ( $s_f$ ) over a designated pruning phase (epochs $t_{start}$ to $t_{end}$ ).
Adaptive Thresholding: At each epoch during the pruning phase, a global magnitude threshold ( $\theta_t$ ) is computed based only on the currently active (non-pruned) weights. This ensures the threshold reflects the evolving weight distribution rather than being skewed by zeroed-out pruned weights.
Monotonic Pruning (No Regrowth): The method enforces a strict monotonic constraint: once a weight is pruned, it remains permanently inactive. The new mask is the intersection of the candidate mask (based on the current threshold) and the previous mask. This eliminates the need for complex connection regrowth mechanisms.
Multi-Stage Mask Enforcement: To ensure pruned weights do not receive gradient updates or drift from zero due to optimizer momentum, masks are enforced at three stages:
1. Pre-Forward: Zeroing weights before the forward pass.
2. Gradient Masking: Zeroing gradients during backpropagation.
3. Post-Optimizer: Re-zeroing weights after the optimizer step to counteract momentum effects.

Training Phases

The training process includes:

Warm-up (Optional): Training without pruning to allow weight magnitudes to stabilize.
Progressive Pruning: Gradual sparsity increase using the linear schedule.
Fine-tuning: Continued training at the target sparsity to optimize the final sparse architecture.

3. Key Contributions

The paper outlines four primary contributions:

Systematic Evaluation: The method is evaluated across ResNet, VGG-style, and LeNet architectures on CIFAR-10 and MNIST, including statistical validation with five random seeds for ResNet-18.
Direct Baseline Comparisons: The authors provide direct comparisons with representative iterative (LTH) and initialization-based (SNIP, GraSP) pruning methods under closely aligned settings.
Simplicity and Effectiveness: The study demonstrates that a linear sparsity schedule with monotonic pruning achieves strong results without complex scheduling functions or weight-regrowth mechanisms.
Sparsity-Accuracy Trade-off Analysis: A detailed analysis across seven sparsity levels (70% to 95%) identifies a favorable operating range where accuracy degradation is minimal.

4. Experimental Results

Experiments were conducted on CIFAR-10 and MNIST using ResNet-18, VGG-like, VGG-19, and LeNet-300-100 architectures.

Performance vs. Baselines

ResNet-18 (CIFAR-10): At 72.9% sparsity, the proposed method achieved 95.12% accuracy, significantly outperforming the reported LTH result of 90.5%, while requiring only a single training cycle.
VGG-like (CIFAR-10): At 97% sparsity, the method achieved 93.13% accuracy, surpassing the reported SNIP result of ~92.0%.
VGG-19 (CIFAR-10): At 97.97% sparsity, the method achieved 93.44% accuracy, outperforming the reported GraSP result of 92.19% at 98% sparsity.
LeNet-300-100 (MNIST): The method outperformed both LTH and SNIP baselines at 78.9% and 98% sparsity, respectively, demonstrating generalization to fully connected networks.

Sparsity-Accuracy Trade-off

Operating Range: For ResNet-18 on CIFAR-10, accuracy remains within 0.1 percentage points of the dense baseline (95.27%) for sparsity levels between 70% and 85%.
High Sparsity: At 90% sparsity, accuracy drops only slightly to 95.06%. A more noticeable degradation occurs at 95% sparsity (93.86%).
Efficiency: At 80% sparsity, the model retains 95.24% accuracy (a 0.03 pp drop) while reducing parameters by 5x (retaining only 20%).

Ablation Study

The study confirmed that the magnitude-based pruning criterion is the most critical factor, as random pruning resulted in a 2.73% accuracy drop. While warm-up and fine-tuning provided marginal gains, the core method remained competitive without them.

5. Significance and Claims

The paper claims that progressive magnitude-based pruning offers a practical, single-cycle alternative to established pruning paradigms. Its significance lies in:

Computational Efficiency: It eliminates the need for multiple training cycles required by iterative methods like LTH.
Adaptability: Unlike initialization-based methods, it allows the pruning mask to adapt as weight importance evolves during training.
Implementation Simplicity: By avoiding connection regrowth and complex scheduling, it reduces implementation overhead compared to dynamic sparse training.
Robustness: The method achieves competitive or superior performance across diverse architectures (ResNet, VGG, LeNet) and extreme sparsity levels (up to ~98%).

The authors conclude that under the evaluated settings, this approach provides a balanced and effective solution for neural network sparsification, making it a viable baseline for practical deployment where computational resources are limited. The paper acknowledges limitations regarding unstructured pruning (which may not yield hardware speedups without sparse compute support) and the need for future testing on larger datasets (e.g., ImageNet) and modern architectures (e.g., Transformers).