Spiking Neural Network Architecture Search: A Survey

1. Problem Statement

The paper addresses the critical challenge of designing optimal Spiking Neural Network (SNN) architectures. While SNNs offer superior energy efficiency and real-time processing capabilities compared to traditional Artificial Neural Networks (ANNs) due to their event-driven, temporal nature, manually designing them is inefficient and often suboptimal.

• Current Limitations: Most existing SNNs are either adapted from ANN architectures (e.g., VGG, ResNet) without considering spike-based dynamics, or designed via manual trial-and-error. Neither approach fully leverages the unique temporal and sparse characteristics of spiking neurons.
• The Gap: There is a lack of automated methods to discover SNN-specific architectures that balance accuracy, latency, and energy consumption while adhering to the strict constraints of neuromorphic hardware (e.g., asynchronous operation, limited precision, and connectivity constraints).
• Core Conflict: Standard Neural Architecture Search (NAS) techniques, designed for differentiable ANNs, fail when applied directly to SNNs due to the non-differentiable nature of spike functions, complex temporal dynamics, and the "hardware/software boundary" where software simulations often misalign with physical neuromorphic chips.

2. Methodology and Framework

The authors propose a comprehensive survey framework for SNN Architecture Search (SNNaS) viewed through a hardware/software co-design lens. The methodology is structured around three core components (illustrated in Figure 1 of the paper):

A. Search Space Design

The paper categorizes how architectural choices are defined, moving beyond simple layer stacking:

• Global/Unstructured: Searching the entire network configuration at once (e.g., MixedSNN, Auto-Spikformer).
• Sequential: Layer-by-layer construction, though often too computationally expensive for SNNs due to temporal dynamics.
• Cell-Based: Defining micro-architectures as Directed Acyclic Graphs (DAGs) of operations (e.g., SNASNet, MA-DARTS). This is currently the dominant approach.
• Hierarchical: Multi-level motifs (micro, meso, macro) mimicking biological structures (e.g., MSE-NAS, ANCoEF).
• Hardware/Neuron-Level: Optimizing specific hardware parameters (PEs, buffer sizes) and neuron parameters (leakage, thresholds, time constants) simultaneously.

B. Search Strategies

The survey evaluates various optimization algorithms adapted for SNNs:

• Evolutionary Methods: Gradient-free approaches (e.g., Genetic Algorithms, CMA-ES) that handle non-differentiable spike functions well.
• Gradient-Based: Using surrogate gradients to approximate the spike function for backpropagation (e.g., SpikeDHS, MA-DARTS).
• Reinforcement Learning (RL): Treating architecture search as a Markov Decision Process (e.g., ANCoEF), effective for multi-objective optimization.
• Bayesian Optimization: Using surrogate models to guide search in expensive design spaces (e.g., SpikeExplorer).
• Hybrid Methods: Combining strategies (e.g., Evolutionary + DRL) to balance exploration and exploitation.

C. Hardware-Awareness & Co-Design

A central methodological contribution is the emphasis on Hardware/Software Co-Exploration.

• Co-Exploration: Instead of fixing hardware and searching for software (or vice versa), frameworks like ANCoEF and ANAS jointly search neural architectures and hardware configurations (e.g., routing, memory mapping).
• Constraints: The search must account for asynchronous timing, limited synaptic weight resolution, and communication overhead between software simulators and physical chips (e.g., Intel Loihi).

D. Acceleration Strategies

To mitigate the high computational cost of training SNNs, the paper details several acceleration techniques:

• Training-Free/Zero-Shot Proxies: Metrics like Sparsity-Aware Hamming Distance (SAHD) or Gram Matrices that rank architectures without full training.
• One-Shot/Supernet: Training a single "supernet" where sub-architectures share weights (e.g., ESNN, AutoSNN).
• Low-Fidelity Evaluation: Using truncated time steps, smaller datasets, or learning curve extrapolation (LCE) to estimate performance early.

3. Key Contributions

The paper makes six primary contributions to the field:

1. Hardware-Software Co-Design Imperative: It establishes that SNNaS cannot be decoupled from hardware constraints. Asynchronous, event-driven hardware requires fundamentally different architectural paradigms than synchronous ANN-centric NAS.
2. Evolution of Search Spaces: It documents the shift from naive ANN adaptations (replacing ReLU with LIF) to SNN-native representations (DAGs, hierarchical motifs, and biologically inspired structures).
3. Temporal Dynamics Integration: It highlights that many current methods underutilize SNNs' defining advantage: time. Future methods must explicitly optimize for spike timing, timestep duration, and temporal attention.
4. Efficiency via Training-Free Evaluation: It showcases how zero-shot metrics and surrogate models have reduced search costs by orders of magnitude, making SNNaS feasible for resource-constrained edge devices.
5. Multi-Objective Optimization: It emphasizes that SNNaS inherently requires balancing competing objectives: accuracy, energy (spike count/SynOps), latency, memory, and spike sparsity.
6. Benchmarking and Reproducibility: It identifies the lack of standardized benchmarks (like SpikeNAS-Bench) and reproducible codebases as a major barrier to progress, calling for hardware-grounded evaluation standards.

4. Results and Findings

• Performance Gains: Co-exploration frameworks (e.g., ANCoEF) consistently outperform sequential optimization methods, achieving up to a 1.81× reduction in Energy-Delay Product (EDP) and 2.73× speedup in search time compared to separate hardware/software optimization.
• Algorithmic Effectiveness:
  • Evolutionary methods are robust for multi-objective problems but computationally expensive.
  • Gradient-based methods (with surrogate gradients) offer fast convergence but risk getting stuck in local minima or failing to capture temporal nuances.
  • Bayesian Optimization is highly data-efficient for small search spaces but struggles with high dimensionality.
• Hardware Constraints: The paper finds that operations like spiking matrix multiplication and max-pooling are problematic on asynchronous hardware due to timing imprecision. Successful architectures often avoid these or use specialized token mixing (e.g., STMixer).
• Metric Validity: While "Spike Count" is a common proxy for energy, Synaptic Operations (SynOps) and Bit-SynOps provide more accurate estimates of hardware cost, especially when quantization is involved.

5. Significance and Future Directions

This survey is significant because it provides the first holistic view of SNNaS, bridging the gap between algorithmic design and neuromorphic hardware implementation. It argues that the future of low-power, real-time AI depends on moving beyond "ANNs with spikes" to SNN-native architectures discovered through co-design.

Future Research Directions identified:

• Improved Temporal Modeling: Integrating temporal attention and optimizing time constants directly within the search space.
• Robust Benchmarks: Developing diverse, hardware-grounded benchmarks that move beyond static image datasets (MNIST/CIFAR) to event-based and temporal tasks.
• Reproducibility: Standardizing tools, codebases, and documentation to prevent the "reproducibility crisis" seen in general NAS.
• Hybrid Optimization: Combining evolutionary, Bayesian, and gradient-based methods to leverage the strengths of each.
• Emerging Hardware: Adapting search strategies for next-generation neuromorphic chips (e.g., Loihi 2, Speck) and in-memory computing paradigms.

In conclusion, the paper posits that SNNaS is not optional but essential for unlocking the full potential of neuromorphic computing, requiring a shift from manual, ANN-centric design to automated, hardware-aware, and temporally-aware architectural discovery.