🤖 machine learning

Event-triggered Implicit Perturbation for Zeroth-Order Fine-Tuning of Spiking Transformers

This paper proposes an event-triggered implicit perturbation architecture (IPZO) for zeroth-order fine-tuning of Spiking Transformers on in-memory computing accelerators, which eliminates costly read-modify-write operations and reduces hardware overhead by generating perturbations only for spike-activated weights while employing an XOR recombination scheme to maintain accuracy comparable to software-based random number generators.

Original authors: Tengteng Lei, Prabodh Katti, Rashi Dutt, Houssem Sifaou, Tan Peng, Osvaldo Simeone, Kai Xu, Bipin Rajendran

Published 2026-08-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tengteng Lei, Prabodh Katti, Rashi Dutt, Houssem Sifaou, Tan Peng, Osvaldo Simeone, Kai Xu, Bipin Rajendran

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, a specific type of computer program known as a transformer has become the dominant force, powering everything from language translation to image recognition. These systems are incredibly powerful but also incredibly hungry for energy and memory, often requiring massive data centers to run. To bring this intelligence to smaller, everyday devices like smartphones or sensors, engineers have turned to a different approach inspired by the human brain: spiking neural networks. Unlike traditional programs that process information continuously, these networks operate on events, firing only when necessary, which makes them naturally efficient. However, teaching these networks to learn new tasks directly on the device has been a stubborn problem. The standard method for training them, which involves calculating how to adjust every single connection in the network, is too complex and memory-intensive for small chips. A simpler alternative exists, one that estimates how to improve the system by making tiny, random nudges to its connections and seeing if the result gets better, but this method has its own severe hardware hurdles that have kept it from being practical.

A team of researchers has now designed a new architecture that clears these hurdles, allowing these efficient brain-like networks to learn directly on specialized computer chips without needing to constantly rewrite their internal memory. The core challenge they faced was that the standard way of making these random nudges requires the chip to repeatedly read its stored information, change it, and write it back. This process, known as read-modify-write, is slow and drains the battery, effectively undoing the energy savings that the specialized chips were designed to provide. Furthermore, generating the vast number of random numbers needed for these nudges for every single connection in the network would require a random number generator so large it would take up more space than the memory itself. The researchers proposed a clever workaround: instead of changing the stored weights directly, they inject the random changes into the calculation process itself, right where the numbers are being added up. This keeps the stored weights untouched and stationary, preserving the chip's efficiency.

To make this work, the team built a specialized unit that only generates these random nudges for the parts of the network that are actually active at any given moment. Because these brain-like networks are naturally sparse, meaning most of their connections are silent at any single instant, this unit only needs to be a fraction of the size of the full network. However, simply reusing the same small set of random numbers for different parts of the network creates a hidden problem: the nudges become too similar to each other, causing the learning process to get stuck or become inaccurate. To solve this, the researchers introduced a mixing scheme that combines these reused numbers in a specific way based on their location, ensuring that every part of the network receives a unique and independent nudge. This mixing process, which they call an address-driven XOR recombination, restores the statistical independence required for the system to learn effectively.

When the team tested this new design, the results were striking. In simulations and hardware tests using a standard chip manufacturing process, their system achieved the same high accuracy as software-based methods that use perfect, independent random numbers. In contrast, a simpler version that just reused the random numbers without the mixing scheme failed significantly, losing nearly ten percentage points of accuracy on image recognition tasks and requiring far more time to learn. The new design also proved to be remarkably energy-efficient. Because it avoids the costly process of rewriting memory, it consumes less than half the energy of the traditional method when running on typical settings. Even though the new mixing unit adds a small amount of extra hardware, the speed at which the system learns means the total energy used over the entire training process is cut by half compared to the simpler reuse method. This work demonstrates that by moving the random adjustments from the storage area to the calculation area and using a smart mixing technique, it is possible to train complex, brain-inspired AI models directly on energy-constrained hardware, opening the door for smarter, more adaptive devices that can learn on the go.

Technical Summary: Event-triggered Implicit Perturbation for Zeroth-Order Fine-Tuning of Spiking Transformers

Problem Statement

The deployment of Spiking Transformers on resource-constrained edge platforms faces significant hurdles when attempting on-chip learning. While Zeroth-Order (ZO) optimization offers a memory-efficient alternative to backpropagation for non-differentiable Spiking Neural Networks (SNNs) by relying solely on forward-pass evaluations, its implementation on In-Memory Computing (IMC) accelerators is hindered by two primary architectural mismatches:

Hardware Overhead of Random Number Generation (RNG): ZO optimization requires statistically independent random perturbations for every weight parameter. Generating these per-weight perturbations using Uniform Random Number Generators (RNGs) incurs a prohibitive area footprint, as the logic required for RNGs (e.g., Linear-Feedback Shift Registers) far exceeds the area of the memory cells storing the weights themselves.
Disruption of Weight-Stationary Dataflow: Conventional Explicit Perturbation ZO (EPZO) requires reading the entire weight matrix, applying perturbations, and writing the modified weights back to the IMC array for every gradient estimation step. These frequent Read-Modify-Write (RMW) operations negate the energy benefits of IMC by reintroducing massive data movement and, in the case of Non-Volatile Memories (NVMs), exacerbate write-endurance limitations.

Existing solutions, such as perturbation-efficient ZO (PeZO) which reuses RNG outputs, reduce hardware overhead but introduce spatial correlations among perturbations, degrading learning accuracy and convergence speed.

Methodology

The authors propose Implicit-Perturbation ZO (IPZO), a hardware-algorithm co-designed architecture that shifts the perturbation injection point from the stored weight domain to the accumulation domain.

Core Architectural Shift

Instead of modifying the weights ( $\theta$ ) stored in the IMC array, IPZO computes the perturbed output by adding the perturbation sum to the standard Matrix-Vector Multiplication (MVM) result. Leveraging the distributivity of MVM, the perturbed output $x \cdot (\theta \pm \epsilon z_i)$ is computed as:
$x \cdot \theta \pm x \cdot \epsilon z_i$
Here, $x \cdot \theta$ is the standard IMC output, and $x \cdot \epsilon z_i$ is a perturbation sum computed by a dedicated Perturbation Generation Unit (PGU). This approach preserves the weight-stationary execution of the IMC array, eliminating repeated RMW operations.

Event-Triggered Perturbation Generation (PGU)

To address the RNG area overhead, the PGU exploits the inherent sparsity of SNNs:

Dimension Reduction: The PGU generates perturbations only for weight rows activated by input spikes at a given time step, rather than for the entire weight matrix. This decouples the RNG array size from the physical dimensions of the IMC array.
Address-Driven XOR Recombination (PGU-XOR): To prevent the statistical correlations caused by reusing a reduced set of RNGs across multiple rows (a problem in the baseline PGU-Reuse scheme), the authors introduce a recombination scheme.
- Spike addresses are decomposed into a quotient ( $q_i$ ) and a remainder ( $r_i$ ).
- Two independent sets of LFSRs generate random numbers: Quotient-associated LFSRs (Q-LFSRs) and Remainder-associated LFSRs (R-LFSRs).
- The final perturbation vector for a specific row is generated by bitwise XORing the selected Q-LFSR output with the corresponding R-LFSR row. This ensures statistically independent perturbations across the entire weight matrix despite the reduced RNG hardware.
Accumulation Network: To handle resource contention when multiple spike addresses map to the same RNG row, an accumulation network serializes access over multiple cycles, trading bounded latency for a compact hardware footprint.

Key Contributions

IPZO Architecture: A novel framework that injects perturbations into the accumulation domain, eliminating repeated RMW operations and preserving weight-stationary IMC execution.
PGU-XOR Scheme: An event-triggered perturbation generator that reduces RNG array dimensions based on spike sparsity and employs address-driven XOR recombination to ensure statistical independence, overcoming the accuracy degradation of simple RNG reuse.
Hardware Validation: A post-layout implementation in TSMC 16-nm CMOS technology demonstrating the feasibility of the design, including area, throughput, and energy analysis.

Experimental Results

The proposed framework was evaluated through algorithmic simulations and hardware implementation:

Accuracy and Convergence:
- On Spikingformer (CIFAR-10), PGU-XOR achieved 76.41% accuracy, matching software RNG baselines (76.53% for Randint), whereas PGU-Reuse degraded to 66.85% (a 9.56 percentage point drop).
- On SpikeGPT (WikiText-2), PGU-XOR achieved a perplexity (PPL) of 54.20, comparable to software baselines (~53.23), while PGU-Reuse resulted in a PPL of 66.01.
- PGU-XOR converged significantly faster, requiring 2.9 $\times$ fewer epochs on CIFAR-10 and 5.7 $\times$ fewer steps on WikiText-2 compared to PGU-Reuse to reach equivalent performance levels.
Statistical Properties: Analysis of the weight update matrices showed that PGU-XOR achieves near-full rank exploration (effective rank ~120), similar to software RNGs, whereas PGU-Reuse collapsed to a low-dimensional subspace (effective rank ~61.5), confirming the restoration of perturbation diversity.
Hardware Efficiency (TSMC 16-nm):
- Area: PGU-XOR incurs a 40.3%–46.0% area overhead relative to PGU-Reuse due to the additional Q-LFSRs.
- Energy: While PGU-XOR has a slightly higher energy cost per Matrix-Vector Multiplication (MVM) (15.2%–48.9% overhead), its faster convergence reduces the total perturbation energy to 0.51 $\times$ that of PGU-Reuse for iso-accuracy tasks.
- Comparison to EPZO: Compared to conventional Explicit Perturbation ZO (EPZO), IPZO with PGU-XOR reduces perturbation energy to 0.46 $\times$ –0.83 $\times$ for a batch size of $B=64$ and $T=4$ time steps. The advantage grows as the product $BT$ decreases, making it particularly suitable for edge devices.

Significance

The paper claims that IPZO provides a viable path for efficient on-chip learning in IMC-based Spiking Transformers. By relocating perturbations to the accumulation domain, the architecture resolves the fundamental conflict between the algorithmic need for independent random noise and the hardware constraints of IMC (specifically the cost of RMW operations and RNG area). The introduction of the PGU-XOR scheme demonstrates that it is possible to reduce RNG hardware overhead via sparsity without sacrificing the statistical independence required for effective gradient estimation. The results suggest that this approach enables adaptive, privacy-preserving edge intelligence by making ZO-based fine-tuning energy-efficient and hardware-feasible on resource-constrained platforms.