🤖 machine learning

Spiking Neural Network inference on FPGAs with hls4ml

This paper presents an extension of the hls4ml toolkit that enables the clock-driven deployment of PyTorch-trained Spiking Neural Networks onto FPGAs, achieving low-latency real-time inference as demonstrated on the Heidelberg Spiking Digits dataset.

Original authors: Barry M. Dillon

Published 2026-06-10

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Barry M. Dillon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Teaching a Digital Brain to "Spit" Instead of "Speak"

Imagine you have a traditional computer brain (an Artificial Neural Network) that talks in a continuous, smooth stream of numbers, like a radio playing a constant song. Now, imagine a different kind of brain (a Spiking Neural Network or SNN) that communicates like a crowd of people in a dark room. Instead of talking, they only make a sound (a "spike") when something specific happens. Between the sounds, they are silent.

This "spiking" method is great for processing time-based information (like audio or sensor data) because it's naturally efficient: if nothing interesting is happening, the brain stays quiet and uses very little energy.

The Problem:
Usually, these "spiking" brains are designed for special, custom-built hardware (like a custom-made musical instrument) that runs on its own unique rules. However, many real-world scientific machines (like particle detectors or medical scanners) already use standard, powerful chips called FPGAs. These chips are like "Lego sets" that engineers can reprogram, but they are used to the "smooth radio" style of computing, not the "spiky crowd" style.

The Solution:
Barry Dillon, the author of this paper, has built a bridge. He updated a popular tool called hls4ml (which acts like a translator) so it can now take a "spiking" brain trained in Python and translate it into instructions that a standard FPGA can understand and run.

How It Works: The Factory Assembly Line

To make this work on a standard chip, the author had to change how the "spiking" brain behaves.

The Clock-Driven Factory:
- Normal Spiking Brain: Works like a chaotic crowd. If someone shouts, the whole room reacts immediately. It's asynchronous (no set schedule).
- This New FPGA Brain: Works like a factory assembly line. Every second, a bell rings (the clock). At the bell, every worker checks their notes, updates their state, and passes a package to the next person. Even if no one shouted, the line still moves.
- Why? Because standard FPGAs are built to run on a strict clock. The author made the spiking brain march in step with this clock so it fits into existing factory workflows.
The "Memory" of the Neuron:
- In a normal computer, a neuron is like a calculator: you give it numbers, it gives you an answer, and then it forgets everything.
- In this spiking brain, the neuron is like a bucket with a leak.
  - When a "spike" (water) comes in, the bucket fills up.
  - If the bucket gets too full (reaches a threshold), it dumps a bucket of water (fires a spike) and resets.
  - But even if it doesn't fire, the water level (the internal state) stays there for the next moment.
- The author's tool keeps track of this "water level" inside the FPGA chip so the brain can remember what happened a few seconds ago.
The "Readout" (The Decision Maker):
- After the brain processes a chunk of data (like a 1.4-second audio clip of a spoken digit), it needs to make a decision.
- The paper tested two ways to decide:
  - Spike Count: Counting how many times the final neurons shouted.
  - Membrane Readout: Checking how full the final buckets are, even if they didn't shout.
- Result: Checking the "fullness" of the buckets (Membrane Readout) turned out to be more accurate than just counting the shouts.

The Experiment: Teaching the Brain to Recognize Digits

The author tested this new system using a dataset called SHD (Heidelberg Spiking Digits).

The Task: The computer listens to recordings of people speaking numbers (0–9) in English and German.
The Input: The audio is converted into a stream of "spikes" (like a Morse code of sound).
The Setup: They built a small, simple brain with one hidden layer of 64 neurons.
The Training: They taught the brain using a standard Python library (snnTorch) and then used a technique called Quantization-Aware Training (QAT). Think of QAT as teaching the brain to do math with a ruler that only has big marks (low precision) instead of tiny millimeter marks (high precision). This prepares the brain to run on the FPGA, which prefers simpler math to save space.

The Results: Fast, Accurate, and Efficient

The paper reports some impressive numbers:

Speed: The FPGA can process one 1.4-second audio clip in about 34 microseconds. That is incredibly fast—faster than a human eye can blink.
Accuracy: Even when they simplified the math (using lower precision) to fit on the chip, the brain still got about 74% accuracy on the test set (recognizing the spoken digits). This is very close to the accuracy of the complex, high-precision version.
Resource Savings: By using lower precision (simpler math), they saved a huge amount of space on the chip. It's like packing a suitcase: if you roll your clothes tightly (low precision), you can fit the same amount of stuff in a much smaller bag.
Fidelity: The chip's behavior matched the computer simulation almost perfectly (over 98% agreement), proving the translation tool works correctly.

What This Means (and What It Doesn't)

What it does: It proves that you can take a modern, "spiking" AI model trained in Python and deploy it onto standard scientific hardware (FPGAs) without needing to build custom, exotic chips. It opens the door for these efficient, time-sensitive brains to be used in real-time scientific instruments.
What it doesn't do (yet):
- It doesn't save power automatically just because the data is sparse. Because the chip runs on a strict clock, it keeps doing the math even when the neurons are silent. The author notes that future updates could add "gates" to stop the math when nothing is happening, which would save power, but that isn't in this paper yet.
- It currently only supports one specific type of spiking neuron (the Leaky Integrate-and-Fire or LIF).
- It is designed for fixed time windows (like a 1.4-second clip), not for infinite, variable-length streams yet.

In summary: The author built a translator that allows "spiky," time-sensitive AI brains to run on standard, clock-driven computer chips. They tested it on a digit-recognition task, and it worked fast and accurately, proving that this new type of AI can be deployed in real-world scientific equipment today.

Technical Summary: Spiking Neural Network Inference on FPGAs with hls4ml

Problem Statement
Spiking Neural Networks (SNNs) offer a naturally temporal machine-learning framework capable of low-latency inference by propagating information through discrete spikes and maintaining internal neuron states. While dedicated neuromorphic processors (e.g., Loihi, SpiNNaker) leverage asynchronous, event-driven architectures to exploit data sparsity for power efficiency, many scientific real-time inference systems rely on conventional synchronous Field-Programmable Gate Arrays (FPGAs). Existing FPGA implementations often require custom, standalone frameworks that do not integrate with standard deep-learning workflows. There is a need for a method to deploy SNNs trained in modern frameworks (like PyTorch) onto standard FPGA firmware using High-Level Synthesis (HLS) without sacrificing the temporal semantics of spiking models, while maintaining the streamlined optimization pipelines available for Artificial Neural Networks (ANNs).

Methodology
This work presents an extension of the hls4ml toolkit to support the clock-driven deployment of SNNs trained in PyTorch (specifically using the snnTorch library) onto FPGA firmware via Vivado/Vitis backends.

Architecture and Workflow: The implementation maps snnTorch modules to hls4ml's intermediate layer representation. The workflow follows the standard hls4ml pipeline: model conversion, C simulation, HLS synthesis, and export. Unlike asynchronous neuromorphic hardware, this approach implements a clock-driven, synchronous execution model where neuron potential updates, dense layer computations, and readout updates are scheduled by HLS pipelines.
Neuron Models: The current implementation supports the Leaky Integrate-and-Fire (LIF) neuron. During conversion, snnTorch Leaky modules are mapped to either LIFNeuron or IFNeuron (when the leak parameter $\beta=1$ ) layers. These layers maintain static internal states (membrane potentials) that persist across top-function calls until a configured readout window is reached.
Readout Mechanisms: A new SNNReadout layer is introduced to handle the decision rule at the end of a fixed time window. Two modes are supported:
1. Membrane Readout: Accumulates the final membrane potentials of the output layer without spiking.
2. Spike-Count Readout: Counts discrete spikes emitted by the final layer over the window duration.
Quantization-Aware Training (QAT): To compress networks for hardware, a custom QAT algorithm is applied to both linear layers and spiking neurons. The training uses surrogate gradients for backpropagation. The QAT forward pass quantizes inputs, weights, biases, neuron parameters, and membrane updates using straight-through estimators, matching the rounding and saturation behavior of the target HLS fixed-point types (e.g., ap_fixed).
Experimental Setup: The system was evaluated using the Heidelberg Spiking Digits (SHD) dataset, which consists of 20 classes of spoken digits encoded as spike trains. The data was preprocessed into 140 timesteps (1.4s window) with 70 pooled input features. A simple network architecture was used: a 70-dim input, a single hidden layer of 64 LIF neurons, and a 20-dim output.

Key Results
The authors validated the design through software reference comparisons, HLS C simulation, and full Vivado synthesis on a Xilinx xczu7ev-ffvc1156-2-e FPGA.

Accuracy and Readout Comparison: Membrane readout SNNs achieved higher accuracy than spike-count readout SNNs (96.79% validation / 75.81% test vs. 94.56% / 73.94% for FP32 models). The membrane readout was selected for subsequent hardware analysis.
Precision and Fidelity: A precision sweep (8 to 24 bits) demonstrated that bit widths of 10 or larger saturate performance, reaching validation accuracies of ~96% and test accuracies of ~74%. The agreement between the HLS C-simulation and the quantized PyTorch reference was extremely high (>98% for bit widths >10), confirming high-fidelity implementation.
Latency and Resources: The synthesized design achieved an inference latency of approximately 33.6 µs per 140-timestep event (15 cycles per timestep $\times$ $\times$ 140 timesteps $\times$ $\times$ 16 ns clock period).
- Resource Utilization: Quantizing the network to ap_fixed<10,4> resulted in significant resource savings compared to higher precisions (e.g., 24-bit) while maintaining accuracy. The 10-bit configuration utilized ~109k LUTs and 0 DSPs, whereas 24-bit required ~63k LUTs and 1,662 DSPs.
Sparsity Metrics: Using NeuroBench metrics, the study found that hidden layer firing is highly sparse (~92% of neurons remain silent per update). However, because the current implementation is clock-driven, these sparsity patterns do not translate to reduced power or latency on the FPGA, as computations are performed regardless of spike activity.

Significance and Claims
The paper claims to open the hls4ml toolkit to neuromorphic computing, providing a streamlined path for optimizing, synthesizing, and deploying SNN models for real-time inference on standard FPGAs.

Integration: It successfully integrates SNN semantics into the existing hls4ml framework, allowing SNNs to be optimized using the same well-tested procedures as ANNs.
Baseline Establishment: The work establishes a practical baseline for deploying SNNs in scientific FPGA workflows, demonstrating that clock-driven SNNs can achieve low-latency inference (~34 µs) with high accuracy on standard hardware.
Limitations and Future Work: The authors modestly note that the current clock-driven approach does not automatically realize the power-saving benefits of sparse data inputs found in asynchronous neuromorphic systems. They identify future work as adding support for recurrent neurons (RLeaky), variable-length sequence boundaries, and gated execution to exploit sparsity for power reduction, as well as expanding applications to anomaly detection and reinforcement learning.

The paper concludes that while the current implementation does not yet match the power efficiency of dedicated neuromorphic hardware for sparse data, it provides a robust, high-fidelity foundation for deploying trained spiking models in time-critical scientific applications using standard FPGA synthesis flows.