Working Memory in a Recurrent Spiking Neural Networks With Heterogeneous Synaptic Delays

This paper proposes an end-to-end trained recurrent spiking neural network that utilizes heterogeneous synaptic delays to encode arbitrary spike patterns as sequential chains of overlapping motifs, achieving perfect recall on a synthetic benchmark and demonstrating the potential for energy-efficient neuromorphic working memory.

The Gist

Imagine your brain is trying to remember a complex melody, like a jazz solo, not just the notes, but the exact rhythm and timing of every single drum hit. For a long time, scientists have struggled to teach artificial computer brains (called Spiking Neural Networks) to do this. These computers are great at recognizing static images (like a cat), but they are terrible at remembering sequences of events that happen over time, especially when there are gaps between them.

This paper presents a clever new way to give these computer brains a "working memory" so they can remember and replay these complex rhythms perfectly.

Here is the breakdown using simple analogies:

1. The Problem: The "Short-Term Memory" Glitch

Think of a standard computer brain like a person with a very short attention span. If you tell them a story, they remember the first sentence, but by the time you get to the third sentence, they've forgotten the first. In technical terms, they can't connect a "spark" (a neuron firing) that happened 5 seconds ago to a "spark" happening right now.

In the real brain, neurons talk to each other using electrical spikes. The problem is that in artificial networks, these spikes usually happen "instantly." If you want to remember a sequence that lasts a long time, the signal gets lost.

2. The Solution: The "Mailman with Different Speeds"

The authors' big idea is to introduce Heterogeneous Delays.

Imagine a post office where every letter (a neural spike) is sent to a friend.

• Old Way: All letters travel at the same speed. If you send a letter today, it arrives today. If you send one tomorrow, it arrives tomorrow. You can't make them arrive together if they were sent at different times.
• New Way (This Paper): Every letter has a different "delivery speed" assigned to it.
  • Letter A is sent today but takes 4 days to arrive.
  • Letter B is sent today but takes 1 day to arrive.
  • Letter C is sent today but takes 3 days to arrive.

Now, imagine you want to trigger a big celebration (a neuron firing) exactly 4 days from now. You can send three different letters today, each with a different delivery speed, so that they all arrive at the recipient's house at the exact same moment on Day 4.

In the computer model, this is called a "Spiking Motif." It's a specific pattern of timing where spikes sent at different times converge perfectly to trigger a new event later.

3. How the Memory Works: The "Domino Chain"

The paper describes a network of 512 neurons. Here is how it remembers a sequence:

1. The Setup: You give the computer a "clamped" start. You force it to fire exactly like a specific pattern for the first 41 milliseconds. Think of this as setting up the first few dominoes by hand.
2. The Chain Reaction: Once you let go, the network has to predict the next domino.
   • Because of the "different delivery speeds" (the delays), the network looks at the spikes it just saw.
   • It calculates: "If I fire neuron X now, and neuron Y fires 3 milliseconds later, they will arrive at neuron Z exactly 10 milliseconds from now."
   • If the timing is right, neuron Z fires.
3. The Loop: That new spike (neuron Z) becomes part of the history for the next step. It helps predict the next spike.
4. The Result: The network creates a self-sustaining chain reaction. It doesn't need to be told what to do next; it just keeps predicting the next beat in the rhythm based on the previous one.

4. The Training: Learning the "Speed Limits"

How did the computer learn which delays to use?

• They didn't just guess. They used a method called Surrogate-Gradient Backpropagation.
• Imagine a teacher grading a student's music performance. The student plays a sequence. The teacher says, "You were a tiny bit late on that drum hit."
• In a normal computer brain, you can't easily say "be a little bit faster" because the timing is digital (on/off).
• But this new system is smart. It adjusts the "speed limits" (the delays) and the "volume" (the weights) of the connections. Over time, it learns exactly how fast each "letter" needs to travel so that the whole orchestra plays in perfect sync.

5. Why This Matters

• Efficiency: This is huge for "Edge AI" (smart devices like hearing aids or robots that run on batteries). This system is incredibly efficient because it only fires when necessary (sparse activity), unlike standard AI which is always "thinking" loudly.
• Biological Realism: Real brains have axons (wires) of different lengths, causing signals to arrive at different times. This model finally embraces that "messiness" and turns it into a superpower.
• The Future: The authors suggest that in the future, we could use this to listen to a real brain (like a recording from a patient with epilepsy) and automatically figure out what patterns the brain is using to remember things, without needing to label the data first.

The Bottom Line

This paper shows that if you give a computer brain a way to send messages at different speeds, it can turn a chaotic stream of noise into a perfect, self-sustaining memory of complex rhythms. It's like teaching a choir to sing a song where every singer starts at a different time, but thanks to their unique walking speeds, they all hit the high note at the exact same moment.

Technical Summary

1. Problem Statement

The paper addresses a fundamental challenge in Spiking Neural Networks (SNNs): implementing working memory. While traditional Artificial Neural Networks (ANNs) use architectures like GRUs or Transformers to handle long-range temporal dependencies, SNNs struggle to bind activity patterns separated by intervals far exceeding a single neuron's time constant.

Previous approaches, such as Synfire chains and Polychronous Neuronal Groups (PNGs), rely on precise spike timing and fixed axonal delays to store information. However, these models face two main limitations:

1. Fixed Delays: Delays are typically hard-coded rather than learned, limiting flexibility.
2. Error Propagation: In sequential tasks, small errors in early time steps compound as the sequence progresses, making it difficult to recall long, arbitrary spike patterns without degradation.

The goal is to develop a biologically plausible, energy-efficient SNN that can learn to store and recall arbitrary temporal spike patterns using learnable heterogeneous synaptic delays.

2. Methodology

Network Architecture: Recurrent HD-SNN

The authors propose a Recurrent Heterogeneous-Delay SNN (HD-SNN) consisting of $N=512$ Leaky Integrate-and-Fire (LIF) neurons.

• Heterogeneous Delays: Unlike standard SNNs with single delays per synapse, each synapse connects to a weight tensor $W \in \mathbb{R}^{N \times N \times D}$, where $D=41$ represents distinct delay channels (0 to 40 ms).
• Spiking Motifs: The network operates by detecting Spiking Motifs (SMs). A motif is defined as a contiguous window of length $D$ where spikes from presynaptic neurons, arriving via specific heterogeneous delays, converge synchronously on a postsynaptic neuron to trigger a spike.
• Recurrent Loop: The network is fully recurrent. The output spikes at time $t$ become the input context for time $t+1$, allowing the network to chain motifs together to form long sequences.

Training Mechanism

• Surrogate-Gradient Backpropagation Through Time (BPTT): Since the spike function is non-differentiable, the authors use surrogate gradients (fast-sigmoid with sharpness $\alpha=15$) to train the network end-to-end.
• Loss Function: The objective is to minimize $L = 1 - F1$, where the F1 score is the harmonic mean of precision and recall between the predicted and target spike trains. This penalizes both over-activation (low precision) and silence (low recall).
• Initialization: Before training, weights are initialized analytically using a Hebbian-like rule (Moore-Penrose pseudo-inverse) based on the cross-correlation of target patterns. This provides a strong starting point similar to Spike-Timing-Dependent Plasticity (STDP).

Experimental Setup

• Dataset: A synthetic benchmark of $M=16$ distinct target spike patterns, each with $N=512$ neurons over $T=1000$ time steps (1 second).
• Memory Cuing: Retrieval is triggered by "clamping" the network to the target pattern for the first $D=41$ steps. The network must then autonomously generate the remaining $T-D$ steps.
• Hardware: Simulations utilized PyTorch with Apple Silicon (M3 Ultra) and NVIDIA CUDA (Jean Zay supercluster).

3. Key Contributions

1. Learnable Heterogeneous Delays: The paper demonstrates that treating delays as learnable parameters within a weight tensor allows the network to dynamically discover the optimal temporal structure for storing sequences, moving beyond fixed-delay models.
2. Chain-of-Motifs Framework: The work generalizes the concept of Spiking Motifs to a fully recurrent topology. It shows that working memory can be implemented as a sequential chain of overlapping temporal predictions, where the context window uniquely predicts the next spike.
3. End-to-End Training of SNNs: The study bridges computational neuroscience (polychronization) and machine learning by successfully training a complex recurrent SNN with surrogate gradients, achieving perfect recall on a challenging benchmark.
4. Efficiency and Sparsity: The model achieves high performance with sparse firing rates ($p_A \approx 10^{-3}$), highlighting its potential for energy-efficient neuromorphic edge deployment.

4. Results

• Perfect Recall: On the synthetic benchmark ($M=16, N=512, T=1000$), the trained network achieved a mean F1 score of 1.0, perfectly reproducing all target patterns.
• Learning Dynamics:
  • Recall accuracy was highest immediately following the clamped initialization window and propagated forward in time.
  • The network successfully avoided pathological attractors (complete silence or over-activation) by optimizing the F1 score, which balances precision and recall.
• Parameter Sensitivity (Figure 4 Analysis):
  • Delay Depth ($D$): Increasing $D$ significantly reduced loss. Larger $D$ provides a larger, more orthogonal context space ($N \times D$), reducing interference between motifs.
  • Pattern Duration ($T$): Loss increased monotonically with sequence length due to error compounding and gradient vanishing, confirming the difficulty of long-horizon prediction.
  • Firing Rate ($p_A$): Performance was optimal at low firing rates ($10^{-4}$ to $10^{-3}$). Higher rates degraded context orthogonality, increasing interference.

5. Significance and Future Directions

• Biological Plausibility: The results support the hypothesis that heterogeneous axonal delays are a computational asset for the brain, enabling the storage of vast amounts of temporal information (working memory) without requiring massive increases in neuron count.
• Neuromorphic Application: The model is designed for hardware compatibility. The ability to store complex sequences with sparse activity and learned delays makes it a strong candidate for deployment on neuromorphic chips for real-time, low-power applications (e.g., event-based vision, brain-machine interfaces).
• Future Work:
  • Unsupervised Learning: Moving from supervised target memorization to self-supervised learning (predicting future spikes from history) to discover latent motifs in raw neural data.
  • Capacity Analysis: Systematically determining the theoretical limits of pattern storage relative to network size and delay depth.
  • Complex Dynamics: Incorporating more sophisticated neuron models (e.g., adaptive thresholds) to enhance intrinsic timescales and retrieval fidelity.

In conclusion, this paper provides a robust, trainable framework for working memory in SNNs, proving that heterogeneous synaptic delays combined with gradient-based learning can solve the problem of long-range temporal binding in spiking systems.