🤖 machine learning

End-to-End Keyword Spotting on FPGA Using Graph Neural Networks with a Neuromorphic Auditory Sensor

This paper presents the first end-to-end FPGA implementation of a keyword spotting system that integrates a Neuromorphic Auditory Sensor with a Graph Neural Network, achieving real-time, low-power processing of raw event-based audio streams with 87.43% accuracy and sub-35 microsecond latency.

Original authors: Wiktor Matykiewicz, Piotr Wzorek, Kamil Jeziorek, Tomás Muñoz, Antonio Rios-Navarro, Angel Jiménez-Fernández, Tomasz Kryjak

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wiktor Matykiewicz, Piotr Wzorek, Kamil Jeziorek, Tomás Muñoz, Antonio Rios-Navarro, Angel Jiménez-Fernández, Tomasz Kryjak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a tiny, battery-powered robot to understand when someone says a specific word, like "Stop" or "Go." Usually, to do this, the robot has to listen to the whole sound, record it, chop it up into tiny pieces, and then analyze those pieces. This process is like trying to read a whole book just to find one specific sentence—it takes a lot of time, energy, and memory.

This paper presents a new, super-efficient way to do this using a special kind of "robot ear" and a "brain" built directly onto a single computer chip (called an FPGA).

Here is how they did it, explained with simple analogies:

1. The "Robot Ear" (The Neuromorphic Sensor)

Most microphones work like a metronome: they take a "snapshot" of the sound 44,000 times every second, whether there is noise or silence. This creates a massive amount of data, even when nothing interesting is happening.

The researchers used a Neuromorphic Auditory Sensor (NAS). Think of this sensor not as a camera taking photos, but as a highly sensitive security guard.

How it works: The guard only raises a hand (sends a signal) when something changes in the room. If the room is silent, the guard sits still. If a bird chirps, the guard raises a hand once.
The Result: Instead of a constant stream of data, the sensor sends a sparse, "event-based" stream. It's like sending a text message only when something important happens, rather than sending a live video feed of an empty room. This saves a huge amount of energy.

2. The "Smart Filter" (Filtration)

Even with the smart sensor, sometimes the "guard" gets a little too excited and raises their hand too many times for the same sound. This creates a lot of unnecessary noise.

The researchers added a Filtration stage. Imagine this as a bouncer at a club.

The bouncer has a rule: "If you've already raised your hand recently, wait a moment before raising it again."
This bouncer cuts out about 47% of the extra signals without losing the important ones. In fact, by removing the noise, the robot actually understood the words better than before.

3. The "Brain" (Graph Neural Network)

Once the signals are filtered, they need to be understood. Usually, computers look at sounds in a straight line (like a timeline). But because these signals are scattered and irregular, the researchers used a Graph Neural Network (GNN).

Think of a GNN not as a straight line, but as a social network map.

Each "event" (the guard raising a hand) is a person in the network.
The system looks at who is standing next to whom and how they are connected in time and space.
Instead of forcing the data into a rigid grid, the system builds a dynamic web of connections to figure out what word was spoken. This is much more flexible and efficient for this type of data.

4. The "All-in-One Chip" (FPGA Implementation)

The biggest achievement of this paper is that they didn't just simulate this on a powerful computer; they built the entire system on a single, small chip (an FPGA).

The Analogy: Imagine building the microphone, the bouncer, and the brain all inside a single, tiny Lego brick.
The Benefit: Because everything is on one chip, the data doesn't have to travel back and forth between different parts of a computer. It stays local. This makes the system incredibly fast and low-power.

The Results: Speed and Efficiency

The team tested this system with a famous list of words (Google Speech Commands). Here is what they found:

Accuracy: It correctly identified words about 87% of the time, even after being simplified to run on the small chip.
Speed: It is lightning fast. From the moment a sound hits the sensor to the moment the chip makes a decision, it takes less than 35 microseconds. To put that in perspective, a human blink takes about 300,000 microseconds. The chip makes a decision in the time it takes for a blink to start.
Power: It uses very little electricity (about 1.12 Watts), which is roughly the power of a small LED light bulb. This makes it perfect for battery-powered robots or IoT devices.

Why This Matters

The paper claims this is the first time a system has successfully combined a neuromorphic sensor and a graph neural network on a single chip to process raw audio in real-time.

Unlike other systems that require heavy pre-processing (like converting sound into complex images before analyzing them), this system handles the raw "events" directly. This means it skips the heavy lifting, saving time and battery life, making it ideal for mobile robotics and smart devices that need to listen and react instantly without draining their batteries.

Technical Summary: End-to-End Keyword Spotting on FPGA Using Graph Neural Networks with a Neuromorphic Auditory Sensor

Problem Statement
Modern embedded systems, particularly in mobile robotics and IoT, require efficient on-device data processing that balances high prediction accuracy with low energy consumption and minimal latency. Conventional audio processing pipelines rely on fixed-rate sampling and computationally intensive digital signal processing (DSP), which often generate redundant data and consume significant power. While neuromorphic sensors offer a bio-inspired alternative by generating sparse, event-based data only when environmental changes occur, integrating these sensors with efficient processing architectures on a single edge device remains a challenge. Prior work often relied on synthetically generated event streams or separated the sensor and processing units, failing to demonstrate a fully integrated, end-to-end system capable of processing raw audio directly from a physical neuromorphic sensor on a single FPGA.

Methodology
The authors propose a unified end-to-end system implemented on a single Field-Programmable Gate Array (FPGA) that integrates a Neuromorphic Auditory Sensor (NAS) with a Graph Neural Network (GNN). The architecture consists of three primary stages:

Neuromorphic Sensing (NAS): The system utilizes a fully digital FPGA-based NAS that converts raw analog audio (via an I2S interface) into a sparse, asynchronous Address-Event Representation (AER) stream. The sensor employs Spike Signal Processing (SSP) principles, converting Pulse Coded Modulation (PCM) samples into high-frequency spike trains via Pulse Frequency Modulation (PFM). The authors implemented two topological configurations:
- Cascade: Chains spike low-pass filters, mimicking biological cochlea but introducing cumulative latency.
- Parallel: Feeds the input spike train into independent spike band-pass filters, eliminating cumulative delay and providing uniform latency across channels.
  The sensor outputs events as tuples $(t, c, p)$ , representing timestamp, frequency channel, and polarity.
Event Filtration: To address the high event rate and uneven distribution across channels, a hardware-friendly filtration stage is introduced. This module mimics a leaky integrate-and-fire (LIF) neuron model, maintaining a decaying potential for each channel. Events are accepted only when the potential exceeds a specific threshold, effectively reducing the event count by approximately 47% while improving prediction accuracy. The filtration parameters include a time quantization factor, a weight for potential integration, and channel-specific thresholds (tested in constant, linear, and exponential configurations).
Graph Generation and GNN Processing: Filtered events are treated as nodes in a dynamic, time-directed graph. Edges are created between a new event and previous events within defined temporal ( $R_t$ ) and channel ( $R_c$ ) radii. The graph is processed by a GNN architecture based on PointNetConv layers, followed by a MaxPool layer that aggregates features within 10 ms windows. The network head consists of linear layers and a GRU-based memory unit to output keyword class probabilities and confidence scores. To ensure real-time performance on hardware, the authors implemented a back-pressure scheduling mechanism with inter-module buffering, allowing dynamic event admission based on instantaneous workload rather than fixed clock cycles.

Key Contributions
The paper makes four primary contributions:

Dataset Release: The authors publicly released event-based audio datasets recorded using multiple NAS configurations (32, 64, and 128 channels in both cascade and parallel modes) derived from the Google Speech Commands v2 (GSCv2) benchmark.
Hardware-Aware Filtration: A novel event filtering method is proposed and analyzed, which reduces the computational load by nearly half while simultaneously enhancing prediction accuracy.
Architectural Optimization: Extensive ablation studies led to specific GNN modifications, including optimized feature extraction and a dynamic scheduling mechanism, enabling the integration of the sensor and network on a single FPGA without violating latency or throughput constraints.
First End-to-End Implementation: The work presents the first system to integrate a physical neuromorphic auditory sensor and a GNN on a single FPGA device for real-time keyword spotting, operating directly on raw audio streams without conventional preprocessing.

Experimental Results
The system was evaluated on the Google Speech Commands v2 dataset processed through the physical NAS.

Accuracy: After 8-bit quantization, the system achieved an accuracy of 87.43% on the 32-channel parallel configuration and 86.9% on the 64-channel parallel configuration.
Latency: The system demonstrated an end-to-end latency of 25–35 µs for the 32-channel configuration and 25–42 µs for the 64-channel configuration. Predictions are generated every 10 ms.
Power Consumption: The average power consumption was measured at 1.12 W for the 32-channel configuration and 1.16 W for the 64-channel configuration.
Resource Efficiency: Compared to previous GNN implementations on SoC FPGAs, the proposed design significantly reduced resource utilization (LUTs, DSPs, and BRAM), allowing the full system to operate at a 200 MHz clock frequency without timing violations.
Comparison: The proposed method outperformed most prior Spiking Neural Network (SNN) and State-Space Model (SSM) approaches reported on similar datasets, achieving higher accuracy with a smaller parameter count (59.84k parameters).

Significance and Claims
The authors claim that this work represents a significant step toward efficient edge intelligence by eliminating the need for conventional signal preprocessing and enabling direct processing of raw audio on a single device. By leveraging the sparsity of neuromorphic data and the efficiency of graph-based processing, the system achieves high temporal resolution (10 ms prediction updates) and low latency, which are critical for mobile robotics and interactive IoT applications. The paper emphasizes that unlike other FPGA-based keyword spotting systems that rely on precomputed features or generate predictions only once per second, this architecture processes raw audio continuously and adapts to instantaneous workload conditions. The authors note that while other works report higher accuracies on standard datasets, those approaches often incur additional preprocessing overhead and latency, whereas this system prioritizes end-to-end efficiency and real-time responsiveness. Future work is planned to validate the approach in real-world robotic human–machine interaction scenarios and to investigate ASIC implementation for further optimization.