💻 computer science

Spiking and Event-driven Neuromorphic Mamba Models for Efficient Speech Recognition

This paper proposes spiking and event-driven neuromorphic variants of the SpeechMamba model that significantly improve activation sparsity and reduce parameters for efficient speech recognition on resource-constrained devices, while introducing a cycle-accurate simulator to facilitate algorithm-hardware co-exploration and further efficiency gains.

Original authors: Tauseef Ahmed, Tao Sun, Jeronimo Castrillon, Kanishkan Vadivel, Guangzhi Tang

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tauseef Ahmed, Tao Sun, Jeronimo Castrillon, Kanishkan Vadivel, Guangzhi Tang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot assistant (like the ones in your phone or smart speaker) that listens to your voice and turns it into text. This robot is incredibly good at its job, but it's also a bit of a glutton. To understand your speech, it constantly crunches numbers, even when there's nothing important to process. It's like a chef who chops vegetables, cooks a meal, and then immediately throws away 90% of the food, only to start chopping again for the next order. This wastes a huge amount of energy and slows the robot down, making it hard to run on small devices like smartphones.

This paper introduces a new way to train these robots so they only "wake up" when they absolutely need to. The authors call this Neuromorphic Computing, which mimics how the human brain works: neurons only fire when there's a strong signal, and they stay quiet otherwise.

Here is the breakdown of their three main tricks, explained simply:

1. The "Smart Threshold" (Event-Driven SpeechMamba)

The researchers took a modern, high-performance speech model called SpeechMamba and taught it to be lazy in a good way.

The Analogy: Imagine a security guard at a club. In the old model, the guard checks every person walking by, even if they are clearly just walking past the door. In the new model, they install a "Smart Threshold." If a person looks like they are just passing by (their signal is too weak), the guard ignores them completely. The guard only processes the people who actually try to enter.
The Result: They used a special mathematical tool called FATReLU to set this threshold. By training the model in three careful stages, they managed to make the robot ignore over 60% of the data it usually processes. The robot still understands speech almost perfectly (losing less than 1% of its accuracy), but it does much less work.

2. The "Binary Switch" (Spiking SpeechMamba)

For the second approach, they went even further by turning the robot's brain into a system of on/off switches.

The Analogy: Instead of the guard measuring how "excited" a person is (a number like 0.5 or 0.8), the guard only sees a light switch: ON (1) or OFF (0). If the switch is OFF, the guard doesn't even look at the person. If it's ON, the guard does a tiny bit of work.
The Result: This "Spiking" model achieved even higher laziness, ignoring over 70% of the data. Interestingly, this model also became smaller, using 30% fewer parameters (memory space) than other similar "spiking" models, while still performing competitively.

3. The "Virtual Test Track" (The Simulator)

The biggest problem with these ideas is that we don't have enough real-life "neuromorphic" hardware (special chips designed for this) to test them properly. Most researchers just guess how much energy they save based on math, which is often wrong because it ignores real-world traffic jams in the computer's memory.

The Analogy: The authors built a video game simulator that acts like a perfect digital twin of a real computer chip. Instead of just guessing how fast the robot would run, they ran the robot inside this simulator to see exactly how many "steps" (cycles) it took and how much "fuel" (memory access) it burned.
The Discovery: The simulator revealed a surprise. The "Binary Switch" model (Spiking) was actually slower in some ways than the "Smart Threshold" model. Why? Because even though the switch was off, the robot still had to constantly check the "battery level" of the switch (membrane potential) to see if it was ready to fire. This extra checking wasted time.
The Fix: Using the simulator, they found the bottlenecks (the traffic jams) and tweaked the "Smart Threshold" model again. This final "Optimized" version saved even more time, improving efficiency by over 10% more than they initially thought.

The Bottom Line

The paper shows that by teaching speech-recognition models to be "lazy" (ignoring unimportant data) and by using a custom simulator to test them on virtual hardware, we can make them much faster and more energy-efficient.

Event-Driven Model: Ignored 60% of data with almost no loss in accuracy.
Spiking Model: Ignored 70% of data and used less memory, but had some hidden "overhead" costs.
The Simulator: Proved that just counting "ignored data" isn't enough; you have to check how the computer actually handles the data to find the real speed bumps.

This work is a step toward having super-efficient AI that can run on your phone or smart home devices without draining the battery or lagging behind.

Technical Summary: Spiking and Event-driven Neuromorphic Mamba Models for Efficient Speech Recognition

Problem Statement

Deep learning has revolutionized Automatic Speech Recognition (ASR), enabling deployment on edge devices like smartphones and smart home systems. However, the dense matrix multiplications inherent in Deep Neural Networks (DNNs) create significant computational and energy burdens, leading to latency and limiting real-time interaction on resource-constrained hardware. While Neuromorphic computing offers a potential solution by introducing activation sparsity through Spiking Neural Networks (SNNs) and event-driven networks, a critical gap remains: there is no comprehensive study evaluating the realistic hardware benefits of different neuromorphic strategies (spiking vs. event-driven) for complex ASR tasks. Existing works often rely on algorithmic metrics (e.g., synaptic operations) that assume ideal sparse computation, failing to capture true hardware costs such as irregular sparsity exploitation and memory access overheads. Furthermore, existing hardware simulators are often too rigid for algorithmic exploration or lack cycle-accurate event-driven modeling for atomic input events.

Methodology

The authors propose adapting the state-of-the-art SpeechMamba architecture (which integrates Mamba blocks with self-attention) into two distinct neuromorphic variants to improve activation sparsity:

1. Event-driven SpeechMamba (E-SpeechMamba)

This approach utilizes FATReLU activation functions to induce sparsity without converting neurons to binary spikes.

Architecture: FATReLU layers are inserted at key bottleneck points (after linear, convolutional, and SSM blocks) within the SpeechMamba encoder and decoder.
Training Pipeline: A three-stage process is employed:
1. Pre-training: Standard ReLU activations are used with a sparsity loss to reduce computational load.
2. Threshold Initialization: ReLU is replaced with FATReLU. Thresholds ( $T$ ) are initialized using a data-driven sweeping strategy, setting $T$ to the mean of the lowest 10% of activation values. Thresholds are progressively increased as long as the normalized loss ratio remains below a tolerance $K$ .
3. Finetuning: Thresholds are further optimized using a sparsity-inducing regularization term ( $L_{spar}$ ) that penalizes large activations and small thresholds, encouraging more zero activations.

2. Spiking SpeechMamba (S-SpeechMamba)

This approach replaces event-driven modules with binary spiking neurons.

Architecture: It incorporates Leaky Integrate-and-Fire (LIF) neurons and adopts the Spikformer attention mechanism (Spiking Self-Attention) and SpikMamba blocks.
Computation: Inputs are projected into query, key, and value spike trains. Attention is computed via scaled dot-product similarity without softmax.
Training: A firing-rate regularization loss ( $L_{quiet} + L_{burst}$ ) is applied to prevent dead neurons (insufficient firing) and excessive firing, balancing sparsity with effective training.

3. Cycle-Accurate Event-Driven Simulator

To bridge the gap between algorithmic metrics and hardware reality, the authors developed a custom simulator:

Platform: Emulates event-driven dataflow execution on a RISC-V Ibex core (RV32IMC), modeling generic digital neuromorphic processors.
Functionality: It performs operation-level simulation where atomic events (spikes or graded activations) trigger immediate partial-sum computations.
Optimization: For SNNs, the simulator skips multiplications by binary zeros and propagates weights for binary ones, bypassing energy-intensive MAC operations.
Usage: The simulator enables "algorithm-hardware co-exploration," allowing the authors to profile submodules, identify bottlenecks, and iteratively optimize the E-SpeechMamba architecture (resulting in E-SpeechMamba (Optimized)).

Key Results

Experiments were conducted on the LibriSpeech dataset (approx. 980 hours of training data).

Performance and Sparsity (Table I)

E-SpeechMamba: Achieved 62% activation sparsity on the test-clean set with less than 1% increase in Word Error Rate (WER) compared to the dense baseline (SpeechMamba).
S-SpeechMamba: Achieved the highest sparsity (72%) but with a higher WER increase (approx. 2.4% on test-clean). Notably, it uses 30% fewer parameters than comparable state-of-the-art SNNs while delivering competitive performance.
E-SpeechMamba (Optimized): Achieved 64% sparsity with similar WER to the base E-SpeechMamba but with greater computation reduction due to additional sparsification points.

Hardware Efficiency (Table II)

Simulated on the RISC-V Ibex core, the models showed significant improvements over a naively sparsified baseline:

E-SpeechMamba (Optimized): Reduced CPU cycles by 46.13%, instructions by 26.9%, memory accesses by 28.50%, and latency by 37.5%.
S-SpeechMamba: Despite higher activation sparsity, it showed lower CPU cycle reduction (19.58%) compared to E-SpeechMamba.
Key Insight: The S-SpeechMamba incurred higher memory access overhead (only 7.63% improvement vs. 17.57% for E-SpeechMamba) due to the cost of maintaining membrane potential states in LIF neurons. This highlights that high activation sparsity does not always translate to hardware efficiency if memory bandwidth is a bottleneck.

Submodule Analysis

Simulation revealed that "hotspots" in the Mamba block (specifically the SSM Scan submodule and certain matrix multiplications) remained dense and dominated execution time. The iterative optimization process, guided by these simulation insights, successfully targeted these bottlenecks to yield further efficiency gains.

Significance and Claims

The paper claims to provide the first study evaluating the realistic hardware benefits of different neuromorphic strategies for complex ASR. Its primary contributions are:

Novel Architectures: Introducing event-driven and spiking variants of the SpeechMamba model that achieve high sparsity with minimal accuracy degradation.
Hardware-Aware Co-Exploration: Demonstrating that algorithmic sparsity metrics alone are insufficient predictors of hardware performance. The authors show that memory access patterns and state maintenance costs (in SNNs) can erode expected gains.
Simulation Framework: Providing a flexible, cycle-accurate, event-driven simulator that allows for the identification of computational bottlenecks and targeted architectural optimization, a capability lacking in existing analytical or coarse-grained simulators.

The work underscores the necessity of hardware-aware analysis to validate neuromorphic methods under realistic conditions and to guide the design of algorithms that can be efficiently supported by future digital neuromorphic hardware.