SpikeDecoder: Realizing the GPT Architecture with Spiking Neural Networks

Technical Summary: SpikeDecoder

Problem Statement
The Transformer architecture, particularly the decoder-only GPT models, has established itself as the dominant paradigm for natural language processing (NLP). However, these models rely on Artificial Neural Networks (ANNs) that perform complex, energy-intensive matrix operations (specifically Multiply-Accumulate or MAC operations) during inference. This results in high energy consumption, limiting their deployment on edge devices or energy-constrained environments. While Spiking Neural Networks (SNNs) offer a promising, energy-efficient alternative due to their event-driven, binary nature, they are notoriously difficult to train directly. Existing SNN-based adaptations of Transformers have largely been restricted to computer vision tasks (e.g., Spikformer) or encoder-only architectures. Furthermore, the few attempts at NLP (e.g., SpikeGPT) have relied on Recurrent Neural Network (RNN) structures like RWKV rather than self-attention, or have utilized ANN-to-SNN conversion rather than direct training. There is currently no fully trainable, spiking-only implementation of a Transformer decoder block for NLP.

Methodology
The authors propose SpikeDecoder, a fully spiking implementation of the Transformer decoder block designed for natural language generation. The development process involved a systematic transition from a standard ANN-based decoder to a purely spiking architecture, analyzing the impact of each component change.

1. Neuron Model: The architecture utilizes the Multi-Step Leaky Integrate-and-Fire (LIF) neuron model, implemented via the SpikingJelly library. This model integrates inputs over time steps, applies a threshold, and resets, mimicking biological neuron dynamics.
2. Embedding Strategies: The paper evaluates various methods to project text characters into spike-compatible formats. Options included one-hot, binary, static (sine/cosine), and learned embeddings. The authors found that concatenating learned token embeddings with static positional encoding offered the best balance of performance and parameter efficiency, though they note that static positional encoding requires specific handling (e.g., lookup tables or Heaviside mapping) to fit binary constraints.
3. Residual Connections: A critical challenge identified was the introduction of floating-point values when adding residual connections between binary spike layers. The authors adopted a restructuring strategy (inspired by Zhou et al., 2023) where LIF neurons are placed immediately after summation operators (both in the Multi-Head Self-Attention and MLP blocks). This ensures that inputs to subsequent linear layers remain binary, eliminating floating-point multiplications caused by residuals.
4. Normalization: Standard Layer Normalization (LN) was found unsuitable for fully spiking models due to its dynamic computation of mean and variance during inference. The authors evaluated Batch Normalization (BN) and Power Normalization (PN). While BN degraded performance, Power Normalization (PN) was identified as a viable alternative that allows for the fusion of linear and normalization layers during deployment, though it incurs higher training times and memory usage.
5. Time-Step Unification: To handle the temporal dimension inherent in SNNs, the authors compared four strategies for aggregating time-step outputs before the classification head: Average, Accumulate, Concatenate, and Final. They determined that Concatenation on spikes was the most suitable for a pure spiking architecture, as it preserves information and produces spike-compatible outputs for the final linear layer.
6. Hybrid Transition: The authors introduced a "Spike Degree" framework (0 to 4) to analyze the transition from ANN to SNN. This allowed them to isolate performance drops at specific stages: embedding conversion, attention mechanism conversion, MLP conversion, and classification head conversion.

Key Contributions

• SpikeDecoder Architecture: The first fully trainable, spiking-only implementation of a Transformer decoder block for NLP, moving beyond encoder-only or vision-based SNN Transformers.
• Systematic Transition Analysis: A detailed breakdown of the performance trade-offs when converting specific Transformer components (embedding, attention, MLP, residuals) to spiking equivalents.
• Component Optimization:
  • Validation of Power Normalization as a viable alternative to Layer Normalization for fully spiking Transformers.
  • Demonstration that learnable LIF thresholds can improve performance (increasing accuracy from ~78% to ~89% in specific configurations), though they noted biological inconsistencies (negative thresholds) in some layers.
  • Identification of Concatenation as the optimal time-step unification strategy for maintaining spike compatibility.
• Energy Efficiency Analysis: A theoretical estimation of energy consumption comparing the SpikeDecoder to its ANN baseline.

Results

• Performance: The fully spiking SpikeDecoder (Spike Degree 4) achieved an accuracy of 87.0% on a character-level prediction task using a subset of "War and Peace," compared to 98.5% for the non-spiking ANN baseline. The transition process revealed that the Multi-Head Self-Attention (SMHA) conversion caused the largest performance drop (approx. 25%), which was partially mitigated by increasing decoder blocks and reducing attention heads.
• Energy Consumption: Theoretical analysis estimated that the SpikeDecoder reduces energy consumption by 86.7% to 92.3% compared to the ANN baseline. This reduction is attributed to the replacement of MAC operations with Synaptic Operations (SOPS) and the elimination of floating-point arithmetic in the core processing blocks.
• Ablation Studies:
  • Removing floating-point additions in residual connections (making them purely binary) caused a catastrophic performance drop (to 18.1%), indicating that current spiking gradients rely heavily on floating-point residuals for stable backpropagation.
  • Using Power Normalization instead of Layer Normalization resulted in a stable accuracy of 81.7% for the fully spiking variant, confirming its feasibility despite longer training times.

Significance and Claims
The paper positions SpikeDecoder not as a state-of-the-art language model surpassing GPT in capability, but as a functional proof-of-concept and a stable starting point for the development of directly trained, spiking Transformer models. The authors modestly claim that their work bridges the gap between the powerful decoder-only Transformer architecture and the energy efficiency of neuromorphic computing. They emphasize that while performance is currently lower than ANN counterparts, the architecture successfully demonstrates that a fully spiking, directly trainable decoder is feasible. The work provides a foundational framework for future research into optimizing spiking NLP models, exploring larger datasets, and refining spike-encoding strategies for natural language.