💻 computer science

Direct-to-Event Spiking Neural Network Transfer

This paper presents the first systematic investigation into converting direct-coded Spiking Neural Networks (SNNs) into energy-efficient event-based representations, addressing key challenges to enable the reuse of pretrained models while preserving performance.

Original authors: Nhan Trong Luu, Duong Trung Luu, Pham Ngoc Nam, Truong Cong Thang

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nhan Trong Luu, Duong Trung Luu, Pham Ngoc Nam, Truong Cong Thang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, high-precision robot (a Spiking Neural Network, or SNN) that was trained in a perfect, simulated world. In this world, the robot sees the world like a standard video camera: every single pixel is a smooth, continuous number (like a shade of gray from 0 to 100). This is called Direct Coding. The robot learned to recognize cats and dogs perfectly using this smooth data.

However, the real world (and the energy-efficient hardware we want to use) doesn't work like a video camera. It works like a Dynamic Vision Sensor (DVS) or a biological eye. These sensors don't see smooth images; they only see events. A pixel only "fires" a tiny electrical spike if something changes (like a leaf moving). If nothing moves, the pixel stays silent. This is called Event-Based Coding.

The Problem: The "Language Barrier"

The paper identifies a major problem: If you take your super-smart robot trained on smooth video data and suddenly force it to look at the world through these "event-only" eyes, it goes completely confused. Its performance crashes.

Why? Because the robot was taught to expect a constant stream of information. When it suddenly gets silence punctuated by tiny, sparse spikes, the math inside its brain breaks. It's like teaching a student to read a novel with full sentences, then handing them a book where only every tenth word is printed, and expecting them to understand the story immediately.

The Goal: The "Translation"

The researchers asked: How do we take a robot trained on smooth data and "translate" it so it can work efficiently with event-based data without losing its smarts?

They call this Direct-to-Event (D2E) Transfer.

The Failed Attempt: "Just Practice" (Task-Specific Finetuning)

The first idea they tried was simple: "Let's just let the robot practice on the event data." They took the trained robot and let it learn from the event data again.

The Result: It helped a little, but the robot still struggled. It was like trying to learn a new language just by listening to it without any guidance. The robot kept making mistakes because the "feel" of the data was so different.

The Solution: "Self-Knowledge Distillation" (SKD)

The authors proposed a clever new method called Self-Knowledge Distillation (SKD). Here is the analogy:

Imagine the robot has two "minds" inside it:

The Teacher Mind: This is the original, smart version trained on smooth video data. It knows the answers perfectly.
The Student Mind: This is the same robot, but it is currently trying to learn using the event-based data.

Instead of just letting the Student guess, the Teacher Mind whispers the answers to the Student. But here's the trick: The Teacher doesn't just say "It's a cat." It says, "It's mostly a cat, but there's a 10% chance it's a dog, and a 2% chance it's a fox."

The Student Mind tries to mimic these probabilities (the "soft" answers) while looking at the event data. By copying the Teacher's "thought process" rather than just the final answer, the Student learns how to interpret the sparse, event-based spikes correctly.

The Results: A Big Win

The paper tested this on various robot "architectures" (different brain structures) using standard image datasets (CIFAR-10 and CIFAR-100) and simulated event sensors.

Performance: The SKD method was a massive success. In many cases, it improved the robot's accuracy on event data by 30% to 50% compared to just "practicing" (finetuning) alone.
Energy: Because the robot is now working with event-based data (spikes), it uses significantly less energy—about 45% less than if it tried to run the old smooth-data method on new hardware.
The Trade-off: The only downside is that the "translation" process (training with the Teacher) takes a bit more computer power and time initially, but the final result is a robot that is both smart and energy-efficient.

The "Why" (The Science Bit)

The paper explains why the robot failed initially using three main reasons:

Information Loss: Converting a smooth image to spikes throws away some data (like compressing a high-res photo into a sketch). You can't get that lost detail back, but you can learn to work with what's left.
Signal Collapse: The "volume" of the signals drops drastically when switching to events. The robot's neurons were set to a loud volume; suddenly, they are hearing a whisper.
Gradient Mismatch: The mathematical direction the robot needs to move to learn is different for the two types of data.

The SKD method acts as a bridge, using the "Teacher's" knowledge to guide the robot through this confusing transition, ensuring it doesn't get lost in the noise.

Summary

In short, this paper solves the problem of taking a smart AI trained on standard data and making it work on ultra-efficient, event-based hardware. They did this by creating a "self-teacher" system that guides the AI through the transition, resulting in a system that is much smarter and more energy-efficient than previous attempts.

Technical Summary: Direct-to-Event Spiking Neural Network Transfer

Problem Statement
Spiking Neural Networks (SNNs) are widely recognized for their potential to enable low-power computation on neuromorphic hardware, particularly when utilizing event-based coding schemes like Time-to-First-Spike (TTFS). However, a significant gap exists between training and deployment. Most high-accuracy SNNs are currently trained using "direct coding," where floating-point input representations are used to simulate continuous-valued surrogate activations via backpropagation. While effective for achieving high precision in simulation, direct-coded SNNs are substantially less energy-efficient upon deployment compared to their event-based counterparts.

The core problem addressed in this work is the lack of a systematic method to convert a high-accuracy SNN pretrained with direct code into an equivalent, energy-efficient event-based representation. Simply retraining from scratch is inefficient, and naive fine-tuning of the pretrained model on event-based data often results in severe performance degradation. The authors frame this as a novel optimization problem: Direct-to-Event (D2E) Transfer.

Methodology
The authors propose a systematic investigation into the D2E transfer problem, comparing two primary strategies:

Task-Specific Finetuning (TSF): This baseline approach treats D2E transfer as a standard domain adaptation problem. It involves taking a pretrained direct-coded SNN and fine-tuning the entire model using event-based inputs and the standard cross-entropy loss. The authors note that while straightforward, TSF lacks theoretical guarantees of optimality for this specific transfer scenario.
Self-Knowledge Distillation (SKD): To address the limitations of TSF, the authors propose SKD, a method grounded in domain adaptation theory and traditional knowledge distillation.
- Theoretical Basis: The authors establish a theoretical bound (Theorem 1) linking the cross-domain accuracy gap between the direct-coded model ( $f_X$ ) and the event-based model ( $f_S$ ) to the Kullback-Leibler (KL) divergence between their output distributions and the total variation distance between the input distributions.
- Mechanism: SKD utilizes the pretrained direct-coded model as a "teacher" to guide the "student" (the same architecture being fine-tuned for event-based inputs). The training objective combines the standard cross-entropy loss on the event-based ground truth with a KL-divergence loss that minimizes the distributional gap between the teacher's direct-coded outputs and the student's event-based outputs.
- Loss Function: The optimization minimizes a weighted sum: $\alpha \cdot L_{CE} + (1-\alpha) \cdot D_{KL}$ , where $\alpha$ balances the ground-truth supervision and the self-distillation signal.

Key Contributions

Problem Formulation: The paper explicitly introduces and defines the D2E transfer problem, distinguishing it from standard finetuning, domain adaptation, and machine unlearning by its focus on preserving the task and label space while shifting only the input representation from direct to event-based.
Theoretical Framework: The authors provide a theoretical derivation (Theorem 1) demonstrating that minimizing the KL divergence between direct and event implementations serves as a proxy for minimizing the cross-domain accuracy gap.
SKD Framework: They propose the SKD method as a practical solution to the D2E problem, offering a principled pathway to transfer representations learned under direct coding to event-based learning.
Comprehensive Evaluation: The study provides the first systematic empirical investigation of this transfer problem across multiple architectures (Spiking-ResNet, Spiking-VGG, Spiking-WideResNet, SEW-ResNet) and datasets (CIFAR-10, CIFAR-100, ImageNet, and simulated Dynamic Vision Sensor data).

Experimental Results
The authors evaluated their methods on standard benchmarks and simulated neuromorphic sensors:

Accuracy Gains: On CIFAR-10 and CIFAR-100, SKD consistently outperformed naive TSF across all tested architectures. For instance, on CIFAR-10, SKD achieved accuracy gains ranging from approximately 25% to 51% over the baseline event-based performance, significantly outperforming TSF. On deeper architectures like Spiking-VGG16, SKD yielded a 51% improvement over the baseline.
Generalization: The method demonstrated robustness on simulated Dynamic Vision Sensor (DVS) data and large-scale ImageNet datasets, consistently achieving the highest validation accuracy compared to TSF.
Energy Efficiency: The transition from direct coding to TTFS coding resulted in a substantial reduction in energy consumption (approximately 45% reduction in millijoules per inference) due to the reduction in synaptic operations (SOPs), validating the energy efficiency goals of the transfer.
Ablation Studies: Experiments varying the loss-balancing coefficient $\alpha$ showed that SKD is robust to hyperparameter choices, with optimal performance typically found between $\alpha = 0.2$ and $0.4$. Pure self-distillation ( $\alpha=0$ ) performed slightly worse than the balanced approach, indicating the utility of ground-truth supervision as an anchor.

Significance and Claims
The paper claims to present the first systematic investigation into the D2E transfer problem. Its primary significance lies in bridging the gap between high-precision SNN simulations (which rely on direct coding) and energy-efficient neuromorphic deployments (which require event-based coding).

The authors assert that conventional finetuning is insufficient for transferring models across heterogeneous input modalities (direct vs. event). In contrast, the proposed SKD framework effectively transfers representations, establishing both a strong empirical baseline and a solid theoretical foundation for future research. The work highlights that while information loss inherent in the encoding map imposes a theoretical ceiling on event-based accuracy, SKD provides a robust mechanism to approach this ceiling while preserving model performance and enabling energy-efficient inference.

The paper acknowledges limitations, noting that SKD introduces additional computational overhead during training compared to TSF and depends heavily on the quality of the pretrained teacher model. However, it positions SKD as a necessary step toward the practical reusability of pretrained SNN databases in energy-sensitive scenarios.