From Lightweight CNNs to SpikeNets: Benchmarking Accuracy-Energy Tradeoffs with Pruned Spiking SqueezeNet

Technical Summary: From Lightweight CNNs to SpikeNets

1. Problem Statement

While Convolutional Neural Networks (CNNs) are the standard for visual recognition, their high energy consumption and computational demands limit their deployment on edge devices and IoT sensors. Spiking Neural Networks (SNNs) offer a promising, energy-efficient alternative due to their event-driven, sparse communication. However, two major gaps exist in current research:

• Scale Disparity: Most SNN research focuses on large-scale models (e.g., VGG, ResNet), leaving the design and evaluation of lightweight SNNs underexplored.
• Performance Gap: SNNs often struggle to match the accuracy of CNNs, particularly when constrained by the resource limitations typical of edge intelligence.

2. Methodology

The authors propose a systematic framework to convert compact CNN architectures into efficient SNNs and optimize them through structured pruning.

• SNN Conversion & Training:
  • Architecture: They converted four lightweight CNNs—ShuffleNet, SqueezeNet, MnasNet, and MixNet—into SNNs by replacing ReLU activations with Leaky Integrate-and-Fire (LIF) neurons.
  • Training Strategy: To overcome the non-differentiable nature of spikes, they used Surrogate Gradient Descent (specifically an arctangent-based function) and a Gradient-Aware Loss function to mitigate vanishing gradients in deep spiking layers.
  • Input Encoding: Pixel intensities were fed as constant currents to maintain deterministic and efficient processing.
• Optimization (Structured Pruning):
  • Identifying SNN SqueezeNet as the most promising base, the authors applied a structured pruning strategy. Instead of removing individual weights, they removed entire redundant "Fire modules" (the core building blocks of SqueezeNet) based on an ablation study to find the optimal balance of depth and efficiency.
• Evaluation Metrics: The models were benchmarked on CIFAR-10, CIFAR-100, and TinyImageNet using accuracy, F1-score, parameter count, MACs (for CNNs), ACs (Accumulate operations for SNNs), and estimated energy consumption (in picojoules).

3. Key Contributions

1. First Systematic Benchmark: The study provides the first comprehensive comparison of lightweight CNN-to-SNN conversions across multiple standard datasets.
2. Identification of Optimal Architecture: The research identifies SqueezeNet as the superior architecture for spiking conversion among the tested lightweight models.
3. SNN SqueezeNet-P: The introduction of a pruned SNN architecture that optimizes the "accuracy-energy" trade-off by removing ineffective modules.
4. Gradient Flow Insights: The work demonstrates that structured pruning can actually improve training by restoring healthier gradient propagation in bottleneck layers.

4. Results

• Superiority of Pruned SNN: The proposed SNN SqueezeNet-P consistently outperformed the baseline SNN SqueezeNet across all datasets. On CIFAR-10, it improved accuracy by 6% and reduced parameters by 19%.
• Accuracy-Energy Trade-off:
  • SNNs achieved up to 15.7× higher energy efficiency than their CNN counterparts (specifically in the case of MixNet).
  • SNN SqueezeNet-P achieved nearly the same accuracy as its CNN version (only 1% lower) while consuming 88.1% less energy (or a 5.6× reduction in the specific comparison provided in Table 2).
• Ablation Findings: The study revealed that late-stage (tail) modules are more critical for discriminative power than early-stage modules. The optimal configuration was found to be a subset of four modules: {F4, F6, F8, F9}.
• Model Comparison: While MixNet was the most energy-efficient, its accuracy was often too low for practical use. SNN SqueezeNet-P emerged as the Pareto-optimal model, sitting on the frontier of the best balance between accuracy and power.

5. Significance

This research establishes a viable pathway for deploying high-performance, low-power intelligence on the edge. By proving that lightweight SNNs can narrow the accuracy gap with CNNs through intelligent architectural pruning, the paper provides practical design guidelines for neuromorphic engineering. It shifts the focus from simply "making models bigger" to "making spike pathways smarter," which is essential for the next generation of always-on, battery-operated edge devices.