💻 computer science

Combining Convolution and Delay Learning in Recurrent Spiking Neural Networks

This paper proposes an enhanced recurrent spiking neural network architecture that integrates convolutional connections with the DelRec delay learning mechanism, achieving a 99% reduction in recurrent parameters and a 52x faster inference speed on audio classification tasks while maintaining the original accuracy.

Original authors: Lúcio Folly Sanches Zebendo, Eleonora Cicciarella, Michele Rossi

Published 2026-04-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lúcio Folly Sanches Zebendo, Eleonora Cicciarella, Michele Rossi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Making "Spiking" Brains Smarter and Faster

Imagine you are trying to teach a computer to recognize spoken words (like "Yes," "No," or "Stop"). To do this efficiently, scientists use Spiking Neural Networks (SNNs). Think of an SNN not as a standard computer program, but as a digital version of a human brain.

Instead of constantly humming with electricity like a standard computer, a brain (and an SNN) only "fires" or "spikes" when it receives a specific signal. This makes them incredibly energy-efficient, perfect for small devices like hearing aids or smartwatches.

However, there's a problem: Time is tricky.
In the real world, sounds happen in a sequence. The word "Cat" isn't just a sound; it's a C sound followed quickly by an a and a t. To understand this, the network needs to remember what happened a moment ago.

The Previous Solution: The "Busy Bee" Network (DelRec)

A few years ago, researchers created a system called DelRec. They realized that in a real brain, signals don't travel instantly; they take a tiny bit of time to travel down the "axon" (the wire connecting neurons).

The Analogy: Imagine a classroom where students (neurons) pass notes to each other. In the old DelRec system, every student had a direct line to every other student in the class.
The Innovation: They added "delays." Some notes took 1 second to arrive, others 5 seconds. This helped the students understand the timing of the conversation.
The Problem: This system was too heavy. If you have 256 students, and every single one talks to every other single one, you have over 65,000 connections. It's like a crowded room where everyone is shouting to everyone else at once. It takes up too much memory and is very slow to process.

The New Solution: The "Local Neighborhood" Network (Conv-DelRec)

The authors of this paper asked a simple question: "Do students really need to talk to everyone in the school, or just their immediate neighbors?"

They realized that in audio signals (like speech), sounds usually have local patterns. The sound of a "K" is closely related to the sound of a "G" or "H" because they come from similar parts of the mouth. They don't need to listen to a sound from a completely different frequency range to understand it.

So, they built a new system called Conv-DelRec.

The Analogy: Instead of a chaotic room where everyone shouts to everyone, imagine the students sitting in a row. Each student only passes notes to the two people sitting directly next to them (left and right).
The Magic: They kept the "delay" feature (the time it takes for the note to arrive).
The Result:
1. Massive Savings: By cutting out the long-distance shouting, they reduced the number of connections by 99%. It's like going from a stadium full of people to a small coffee shop.
2. Super Speed: Because there is so much less "noise" to process, the network runs 52 times faster.
3. Same Smarts: Surprisingly, even though they stopped the students from talking to everyone, the network is just as good at recognizing words as the old, heavy version.

Why This Matters

It's Efficient: You can now put this "brain" on tiny, battery-powered devices (like a smartwatch or a medical sensor) without draining the battery.
It's Fast: It can process speech in real-time, making it great for live translation or voice commands.
It's Scientifically Cool: They proved that you don't need a "super-connected" brain to understand time. A "local neighborhood" brain with good timing (delays) is enough.

The "Aha!" Moment: Learning the Delays

One of the most important parts of the paper is that the network learns how long the delays should be.

Fixed Delay: Imagine a teacher saying, "Everyone, wait exactly 2 seconds before speaking." This is rigid and often wrong.
Learnable Delay: The network figures out, "Hey, for this specific word, I need to wait 3 seconds, but for that one, I only need 1."

The researchers tested this by forcing the network to use fixed delays, and the performance dropped significantly. This proves that the ability to adjust the timing is the secret sauce that makes these networks so good at understanding speech.

Summary in One Sentence

The authors took a heavy, slow, over-connected brain model, trimmed it down to a lightweight, local neighborhood model, and kept the "timing" feature, resulting in a system that is 99% smaller, 52 times faster, and just as smart as the original.

1. Problem Statement

Recurrent Spiking Neural Networks (RSNNs) are powerful for modeling temporal dependencies in sequential data (e.g., speech, biomedical signals) due to their event-driven nature and biological plausibility. However, existing state-of-the-art RSNNs face two critical challenges:

Scalability and Parameter Overhead: Traditional RSNNs, such as the recent DelRec architecture, utilize fully dense recurrent connections. This results in a quadratic parameter growth ( $N^2$ ) relative to the number of neurons ( $N$ ). For a layer with 256 neurons, this creates over 65,000 recurrent parameters, making the model memory-intensive and computationally expensive, especially for edge devices.
Inefficiency in Temporal Modeling: While dense connections capture global correlations, many temporal signals (like audio spectrograms) exhibit strong local correlations (e.g., adjacent frequency bands). Dense connectivity is often redundant for these signals, leading to inefficient resource usage.
Training Stability: Training RSNNs over long time sequences often suffers from vanishing or exploding gradients. While learnable delays (axonal delays) have been shown to mitigate this by acting as temporal skip connections, their integration into scalable architectures remains an open challenge.

2. Methodology

The authors propose a novel architecture that replaces the dense recurrent matrix of DelRec with 1D Convolutional Recurrent Connections while retaining the Learnable Axonal Delays mechanism.

Core Components:

Convolutional Recurrence: Instead of a dense weight matrix $W^{rec} \in \mathbb{R}^{N \times N}$ $W^{r ec} \in R^{N \times N}$ , the authors introduce a lightweight 1D convolutional kernel $W^{conv} \in \mathbb{R}^k$ $W^{co n v} \in R^{k}$ (where $k=3$ $k = 3$ ).
- Each neuron receives recurrent input only from itself and its immediate neighbors (local connectivity).
- This exploits the local correlation structure in audio spectrograms (where adjacent frequency bands are highly correlated).
- Implementation is realized via 2D convolution (kernel shape $k \times 1$ ) on the neuron dimension to integrate with existing buffer mechanisms.
Learnable Axonal Delays: The architecture retains the DelRec mechanism where each neuron $j$ $j$ has a learnable delay $d_j$ $d_{j}$ .
- A differentiable triangular spread function $h_{\sigma, d}(\tau)$ is used during training to allow gradients to flow through the discrete delay values.
- During inference, delays are rounded to the nearest integer.
- This mechanism acts as a temporal skip connection, helping to capture long-range dependencies and stabilize training.
Network Architecture: The model consists of multiple hidden layers. Each layer includes:
1. Feedforward linear projection.
2. Batch Normalization.
3. Leaky Integrate-and-Fire (LIF) neurons.
4. Convolutional Recurrent Delay Unit (combining the 1D convolution and delay buffer).
5. Dropout.
- The output layer uses a non-spiking readout (integrating membrane potentials) followed by a Softmax classifier.

3. Key Contributions

Convolutional Recurrent Architecture: The introduction of 1D convolutions to replace dense recurrent layers in DelRec, drastically reducing the parameter count from $O(N^2)$ to $O(k \cdot N)$ (where $k$ is the kernel size).
Efficiency without Accuracy Loss: Demonstration that local connectivity combined with learnable delays is sufficient for high-performance temporal modeling, challenging the necessity of dense recurrence for audio tasks.
Ablation on Delay Learning: A rigorous study proving that learnable delays are critical. Fixed delays (even at optimal mean/median values) result in significant accuracy drops, confirming that adaptive temporal dynamics are essential for capturing complex dependencies.
Scalability: The proposed method scales linearly with layer width, making it suitable for wider networks and resource-constrained neuromorphic hardware.

4. Experimental Results

The method was evaluated on two neuromorphic audio benchmarks: Spiking Heidelberg Digits (SHD) and Spiking Speech Commands (SSC).

SHD Dataset (Digits 0-9):

Accuracy: The proposed 4-layer Conv-DelRec achieved 91.51% accuracy, nearly matching the original 4-layer DelRec (90.41%) and the 2-layer DelRec (91.72%).
Parameter Reduction: A massive 99.6% reduction in recurrent parameters (from ~65,792 to 259 per layer).
Inference Speed: 52× faster inference time (1.51 ms vs. 38.03 ms for the 4-layer dense model).
Stability: Lower standard deviation in accuracy (0.70% vs 0.84%) indicates more stable training dynamics.

SSC Dataset (35 Words):

Accuracy: Achieved 78.59% accuracy using baseline hyperparameters. While slightly lower than the original DelRec (82.58%), the trade-off in efficiency is significant.
Speed: Inference time reduced from 112.64 ms to 4.19 ms (~27× faster).

Delay Statistics:

The learned delays in the convolutional model showed broader distributions (higher standard deviation) compared to the dense DelRec model, suggesting a richer, more heterogeneous allocation of temporal scales across neurons.

5. Significance and Impact

Edge AI Deployment: By reducing the memory footprint by ~99% and accelerating inference by over 50×, this work makes high-performance RSNNs viable for resource-constrained edge systems and neuromorphic hardware.
Paradigm Shift in RSNN Design: The results suggest that for signals with local spatial structure (like audio), dense recurrence is an unnecessary bottleneck. Local connectivity is sufficient when paired with sophisticated temporal mechanisms like learnable delays.
Training Stability: The study reinforces the hypothesis that learnable delays act as effective temporal skip connections, mitigating gradient issues and improving the network's ability to model long-range dependencies without increasing neuron complexity.

In conclusion, this paper successfully bridges the gap between biological plausibility (learnable delays) and engineering efficiency (convolutional sparsity), offering a scalable, high-performance alternative to dense Recurrent Spiking Neural Networks.