A flexible framework for structural plasticity in GPU-accelerated sparse spiking neural networks

1. Problem Statement

While most research in Artificial Neural Networks (ANNs) and Spiking Neural Networks (SNNs) focuses on synaptic plasticity (adjusting connection strengths), biological brains rely heavily on structural plasticity (creating and removing connections) for learning, adaptation, and recovery.

• Limitations of Current ML: Standard machine learning frameworks (PyTorch, JAX) are optimized for dense connectivity. While pruning is used to reduce inference costs, it does not reduce training costs because the underlying dense matrix structures remain during training.
• Simulation Bottlenecks: Existing structural plasticity implementations are largely restricted to CPU-based simulators or specific neuromorphic hardware (e.g., SpiNNaker). Simulating structural changes on GPUs is computationally expensive due to:
  1. The need to re-allocate memory data structures when connections change.
  2. The difficulty of moving large amounts of data during updates.
  3. The challenge of maintaining efficient, coalesced memory reads required for GPU parallelism.
• Gap: There is a lack of a flexible, efficient framework for implementing structural plasticity rules on general-purpose GPUs that supports both machine learning training and computational neuroscience modeling.

2. Methodology

The authors extend the GeNN (GPU-accelerated Neural Network) simulator to support structural plasticity. GeNN is chosen for its ability to handle sparse connectivity using "ragged matrix" data structures.

Core Technical Framework: Custom Connectivity Updates

The authors introduce a new primitive called "custom connectivity updates" within GeNN. This allows users to define hybrid CPU/GPU algorithms for modifying network topology without re-allocating the entire data structure.

• Data Structure: Connectivity is stored as a ragged matrix (2D array where rows represent presynaptic neurons and columns represent postsynaptic targets, padded to the maximum out-degree).
• Update Mechanism:
  • No Re-allocation: Connections are added to the end of a row or replace existing entries; removal is handled by swapping the target entry with the last entry in the row and reducing the out-degree count.
  • Hybrid Execution:
    • Host (CPU) Code: Handles serial tasks like distributing rewiring attempts across neurons or summing global statistics (e.g., total "dormant" synapses).
    • Device (GPU) Code: Parallelizes updates across rows (presynaptic neurons). It uses bitfields to track potential connection slots and atomic operations for safe updates.
  • Bitfields: Auxiliary data structures track the "sign" of weights (for DEEP R) or the presence of candidate connections (for topographic maps) to avoid duplicate checks.

Application 1: Supervised Learning with DEEP R

• Goal: Train sparse recurrent SNN classifiers with high efficiency.
• Algorithm: DEEP R (Dormant Elimination and Formation of Random connections).
  • Operates alongside gradient-based learning (e-prop).
  • Elimination: If a weight update would flip the sign of a connection, the synapse is marked "dormant" and removed.
  • Formation: An equal number of new random synapses are added to replace the removed ones.
  • Constraint: Maintains constant sparsity throughout training.
• Implementation: Integrated into mlGeNN (a Keras-like abstraction for GeNN) to allow easy definition of sparse networks.

Application 2: Unsupervised Topographic Map Formation

• Goal: Simulate the self-organization of neural maps (e.g., retinotopic maps).
• Algorithm: A variation of the Bamford, Murray, and Willshaw model.
  • Formation: New connections are probabilistically formed based on the Euclidean distance between neurons (Gaussian dependence).
  • Elimination: Weak synapses (low weight) are more likely to be removed than strong ones (Activity-dependent via STDP).
  • Parallelism: Unlike previous sequential implementations, this framework distributes rewiring attempts across all presynaptic neurons in parallel on the GPU.

3. Key Contributions

1. First GPU Structural Plasticity Framework: The first general-purpose framework for implementing structural plasticity on GPUs, overcoming the memory re-allocation bottleneck.
2. Flexible Syntax: Extends GeNN's code-generation approach, allowing users to define complex plasticity rules using C99-like strings (GeNNCode) for both host and device execution.
3. Efficient Sparse Training: Demonstrates that structural plasticity can be applied during training with negligible overhead (<1% for DEEP R), enabling the training of extremely sparse networks that match dense network performance.
4. Scalable Topographic Modeling: Successfully scales topographic map simulations to larger network sizes (up to 12,544 neurons) with faster-than-realtime performance, despite the high computational cost of frequent connectivity updates.

4. Results

Classification Performance (DEEP R)

• Datasets: N-MNIST and DVS Gesture.
• Accuracy: Sparse models trained with DEEP R (5% input, 1% recurrent connectivity) achieved accuracy comparable to dense baseline models (e.g., 88.86% vs. 88.94% on DVS Gesture).
• Efficiency:
  • Parameter Reduction: Achieved similar accuracy with 90× fewer parameters (60k vs. 5.5M).
  • Training Speed: Sparse models trained 8× to 15× faster than dense counterparts.
  • Overhead: The DEEP R update mechanism added less than 0.5% to the total training time.

Topographic Map Formation

• Dynamics: The model successfully refined receptive fields, with neurons aligning their preferred locations to the stimulus center.
• Scaling: Simulations ran faster-than-realtime even at the largest tested scale (s=7, 12,544 neurons).
• Performance Bottleneck: While neuron updates were fast, connectivity updates (specifically the "remapping" of the transpose data structure) accounted for ~90% of the runtime in the topographic model. However, the parallel nature of the GPU implementation allowed for significant speedups compared to previous CPU/SpiNNaker implementations.

5. Significance and Future Directions

• Bridging Biology and ML: The framework enables the study of biologically plausible structural plasticity mechanisms (like neurogenesis and rewiring) at scales previously impossible on GPUs.
• Efficient AI: It validates the "Lottery Ticket Hypothesis" in SNNs, showing that sparse networks can be trained from scratch to match dense performance, drastically reducing memory and compute requirements.
• Neuromorphic Applications: The framework is applicable to neuromorphic hardware (e.g., Loihi, BrainScaleS-2) and other code-generation simulators (Brian 2, NESTML).
• Future Work: The authors suggest exploring guidance mechanisms for rewiring (using gradients to guide where new synapses form) and combining structural plasticity with other learning rules like Eventprop.

In summary, this paper provides a critical infrastructure advance for the field, moving structural plasticity from a CPU-bound, slow simulation concept to a high-performance, GPU-accelerated tool for both biological modeling and efficient machine learning.