Training Deep Normalization-Free Spiking Neural Networks with Lateral Inhibition

1. Problem Statement

Spiking Neural Networks (SNNs) are promising for neuromorphic computing due to their energy efficiency and biological plausibility. However, training deep SNNs faces a critical trade-off:

• Performance vs. Biological Realism: High-performance deep SNNs typically rely on explicit normalization schemes (e.g., Batch Normalization, BN) borrowed from Artificial Neural Networks (ANNs). While effective for stability, these schemes are biologically implausible as they rely on batch statistics that do not exist in biological brains.
• Training Instability: Approaches that prioritize biological realism (e.g., using distinct Excitatory/Inhibitory populations) often fail to train deep networks due to instability, leading to neuron silencing or saturation.
• The Gap: There is a lack of a biologically grounded mechanism that can replace Batch Normalization to stabilize the training of deep SNNs while maintaining the Excitatory-Inhibitory (E-I) dynamics observed in the cerebral cortex.

2. Methodology

The authors propose DeepEISNN, a normalization-free learning framework that replaces standard feedforward layers with a biologically inspired E-I Circuit. This circuit mimics cortical interactions by separating neurons into distinct Excitatory (E) and Inhibitory (I) populations.

2.1 The E-I Circuit Architecture

• Population Segregation: Each layer contains separate populations of excitatory and inhibitory neurons (ratio 4:1, based on biological evidence).
• Sign Constraints: All outgoing weights from a single neuron must share the same sign (positive for E, negative for I), adhering to Dale's Law.
• Lateral Inhibition Mechanism: The circuit regulates activity through two types of inhibition:
  1. Subtractive Inhibition: Balances the mean input to prevent saturation or silence (E-I balance).
  2. Divisive Inhibition: Controls the gain (variance) of the input, acting similarly to normalization.
• Neuron Models:
  • Excitatory Neurons: Modeled using Leaky Integrate-and-Fire (LIF) dynamics.
  • Inhibitory Neurons: Modeled as "fast-spiking" units. Due to their small membrane time constants ($\tau_I \ll \Delta t$), they are approximated as stateless ReLU-like units that reach a steady state instantly within a time step.

2.2 Key Techniques for Stability

To enable stable end-to-end training without normalization, the authors introduce two core components:

A. E-I Init (Dynamic Parameter Initialization)
Standard zero-centered initialization (Xavier/Kaiming) fails under sign constraints. E-I Init ensures a stable start by:

• Balancing Inputs: Setting expected subtractive inhibitory currents to approximately equal expected excitatory currents, resulting in a near-zero net input (preventing saturation).
• Gain Control: Setting the divisive inhibition strength such that the expected divisive current approximates the standard deviation of excitatory inputs.
• Dynamic Estimation: Instead of assuming fixed firing probabilities, the method uses the first training batch to dynamically estimate statistics (mean, variance) to initialize weights ($W_{EE}, W_{IE}, W_{EI}$) and gain parameters ($g_I$).

B. E-I Prop (Stabilizing End-to-End Training)
This toolkit addresses instabilities arising from the forward and backward passes:

• Adaptive Stabilization (Forward Pass): To prevent division-by-zero errors in the divisive inhibition term, the method dynamically replaces zero denominators with the smallest positive value found within the same sample (rather than a fixed $\epsilon$). This preserves the dynamic range and gain control properties.
• Straight-Through Estimator (STE) (Backward Pass): Since the adaptive replacement is non-differentiable, STE is used to bypass the operation during backpropagation, treating it as an identity function to ensure clean gradient flow.
• Gradient Scaling: Theoretical analysis shows gradients for lateral inhibitory weights ($W_{EI}$) are disproportionately large due to the divisive operation. The authors scale these gradients by $1/d$ (where $d$ is input dimension) to balance updates between feedforward and lateral pathways.

3. Key Contributions

1. Normalization-Free Framework: Introduced a deep SNN architecture that achieves stable training without Batch Normalization by incorporating canonical E-I circuits with lateral inhibition.
2. E-I Init Scheme: Developed a dynamic initialization strategy that establishes E-I balance and gain control, solving the "dead neuron" problem in deep E-I networks.
3. E-I Prop Toolkit: Proposed a combination of adaptive stabilization and STE to handle numerical instability and non-differentiable operations, enabling robust gradient flow.
4. Biological Insight: Demonstrated that the E-I circuit learns a bimodal distribution of integrated currents (separating activated and suppressed populations), which is functionally distinct from the Gaussian-like distributions produced by standard BN.

4. Experimental Results

The framework was evaluated on static (CIFAR-10/100, TinyImageNet) and neuromorphic (CIFAR10-DVS, DVS-Gesture) datasets.

• Performance:
  • CIFAR-10: Achieved 92.05% accuracy with ResNet-18, outperforming all other normalization-free baselines and competing with BN-equipped models.
  • TinyImageNet: Achieved 50.29% accuracy, demonstrating scalability to large-scale datasets.
  • Neuromorphic: Outperformed several BN-based methods on DVS-Gesture (94.86%) and CIFAR10-DVS (77.66%).
• Ablation Studies:
  • Removing E-I constraints or using standard initialization led to training failure or significant accuracy drops.
  • Replacing adaptive stabilization with fixed $\epsilon$ caused convergence failure.
  • Removing gradient scaling caused training collapse.
• Comparison: The method significantly outperformed previous E-I constrained models (e.g., DANN, BackEISNN), with the performance gap widening as network depth increased.

5. Significance and Impact

• Bridging AI and Neuroscience: This work provides a practical solution for training deep SNNs that are both high-performing and biologically plausible, eliminating the need for artificial normalization layers.
• Computational Platform: It offers a new computational model for exploring how cortical E-I interactions perform complex, large-scale computations, revealing that E-I dynamics can learn sophisticated activity regulation distinct from standard normalization.
• Hardware Compatibility: The framework is well-suited for next-generation analog or mixed-signal neuromorphic hardware (e.g., DYNAP-SE2), which natively supports such operations, unlike digital chips that struggle with the approximations required for BN.
• Future Direction: It opens the door for deeper, more realistic brain-inspired models that can serve as in silico platforms for investigating multi-scale cortical computation.

Code Availability: The source code is publicly available at https://github.com/vwOvOwv/DeepEISNN.