Energy-Aware Spike Budgeting for Continual Learning in Spiking Neural Networks for Neuromorphic Vision

This paper proposes an energy-aware spike budgeting framework that integrates experience replay, learnable neuron parameters, and an adaptive scheduler to effectively mitigate catastrophic forgetting while optimizing both accuracy and energy efficiency in Spiking Neural Networks across diverse frame-based and event-based neuromorphic vision benchmarks.

The Gist

Imagine you have a super-efficient, battery-powered robot eye (a Spiking Neural Network) that sees the world not like a normal camera taking 30 photos a second, but like a human eye: it only notices when something moves or changes. This makes it incredibly energy-efficient, perfect for tiny devices that need to run for years on a single battery.

However, there's a big problem: Catastrophic Forgetting.

The Problem: The Robot with a Short Memory

Imagine teaching this robot to recognize a cat. It learns perfectly. Then, you teach it to recognize a dog. Suddenly, it forgets what a cat is and starts calling everything a dog. This is "catastrophic forgetting." In the world of AI, when a system learns new things, it often overwrites the old things.

Existing solutions for regular computers (Artificial Neural Networks) try to fix this by "freezing" old memories or making the robot practice old tasks. But these solutions are heavy, energy-hungry, and don't work well with the unique, "event-based" way our robot eye sees the world.

The Solution: The "Energy Budget" Manager

The authors of this paper propose a clever new system called Energy-Aware Spike Budgeting.

Think of the robot's brain as a busy city. Every time a neuron (a brain cell) fires, it sends a tiny electrical "spark" (a spike).

• Too many sparks? The city gets chaotic, wastes battery, and the robot gets confused (overfitting).
• Too few sparks? The city is too quiet, the robot can't hear the important signals, and it misses details (underfitting).

The authors built a Smart Traffic Controller (the "Spike Scheduler") that manages how many sparks are allowed to fly, but it does something amazing: it changes its rules depending on what the robot is looking at.

Analogy 1: The Busy Highway vs. The Quiet Country Road

The paper discovers a "duality" (two opposite behaviors) based on the type of camera the robot uses:

1. The "Frame-Based" Camera (Like a standard video camera):

   • The Situation: This camera sends a constant stream of data, even when nothing is moving. It's like a busy highway where cars (spikes) are everywhere, causing traffic jams.
   • The Fix: The Smart Traffic Controller acts like a police officer. It says, "Okay, we have too many cars! Let's close some lanes and reduce the speed limit."
   • The Result: By forcing the robot to be more efficient (fewer sparks), it actually learns better. It stops getting distracted by redundant noise. On simple tasks like recognizing handwritten numbers, this cut energy use by nearly half while making the robot smarter.

2. The "Event-Based" Camera (Like a motion-sensor eye):

   • The Situation: This camera only sends data when something moves. It's like a quiet country road where cars rarely pass.
   • The Fix: Here, the Smart Traffic Controller acts like a generous host. It says, "It's too quiet! We need more cars to make sure we don't miss the important delivery." It relaxes the rules and allows a few more sparks to fly.
   • The Result: On complex tasks like recognizing hand gestures, allowing a tiny bit more activity helped the robot understand the timing of the movements much better. It boosted accuracy by a huge 17% while still staying incredibly energy-efficient.

How It Works (The Secret Sauce)

The system uses three main tools working together:

1. The Memory Box (Experience Replay): Just like a student reviewing old flashcards before a new exam, the robot keeps a small, balanced box of old examples. It practices these occasionally so it doesn't forget them.
2. The Adjustable Brain Cells (Learnable Dynamics): Standard robot brains have fixed settings. This robot can learn how fast or slow its neurons should react. It's like giving the robot the ability to adjust its own "thinking speed" depending on whether the task is fast (a quick gesture) or slow (a static image).
3. The Budget Manager (The Scheduler): This is the star of the show. It constantly checks: "Are we using too much energy? Or too little?" It automatically tightens or loosens the rules to keep the robot in the "Goldilocks zone"—not too lazy, not too frantic.

The Big Takeaway

The paper proves that energy isn't just a side effect; it's a tool.

By treating energy consumption as a primary goal (like accuracy), the researchers created a robot that:

• Remembers better: It doesn't forget old tasks when learning new ones.
• Thinks smarter: It knows when to be frugal (on busy data) and when to be generous (on sparse data).
• Runs longer: It uses significantly less battery, making it perfect for real-world devices like smart glasses, drones, or medical implants.

In short, they taught the robot to manage its own energy budget so intelligently that it became both smarter and more efficient, solving the "forgetting" problem in a way that works perfectly for the next generation of low-power AI.

Technical Summary

Here is a detailed technical summary of the paper "Energy-Aware Spike Budgeting for Continual Learning in Spiking Neural Networks for Neuromorphic Vision."

1. Problem Statement

The paper addresses the critical challenge of catastrophic forgetting in Spiking Neural Networks (SNNs) when deployed in continually evolving environments. While SNNs offer ultra-low-power perception suitable for neuromorphic hardware, existing continual learning (CL) methods—mostly designed for dense Artificial Neural Networks (ANNs)—fail to jointly optimize accuracy and energy efficiency.

Key gaps identified:

• Modality Mismatch: Most methods do not account for the fundamental difference between frame-based data (dense, Poisson-encoded) and event-based data (naturally sparse, asynchronous).
• Energy Neglect: Existing approaches rarely treat energy consumption as a primary optimization objective, often assuming sparsity is implicitly handled or ignoring the trade-off between spike rates and accuracy.
• Rigid Constraints: Applying uniform sparsity constraints across different modalities can lead to underfitting on event-based data or overfitting on frame-based data.

2. Methodology

The authors propose an Energy-Aware Spike Budgeting Framework that integrates three core components to manage the plasticity-stability trade-off in SNNs:

A. Adaptive Spike Scheduler (Proportional Control)

The core innovation is a feedback control mechanism that dynamically regulates network activity during training.

• Mechanism: It treats spike budgeting as a control problem. A regularization coefficient ($\lambda_{rate}$) is adjusted based on the error between the current spike rate ($r_{spike}$) and a target budget ($r_{target}$).
• Loss Function: The total loss includes a task loss and a squared penalty term: $L_{total} = L_{task} + \lambda_{rate} \cdot (r_{spike} - r_{target})^2$.
• Bidirectional Gradient:
  • If $r_{spike} > r_{target}$, the penalty increases, suppressing activity (enforcing sparsity).
  • If $r_{spike} < r_{target}$, the penalty decreases, allowing the task loss to drive activity up (relaxing constraints).
• Modality Adaptation: This allows the system to enforce sparsity on dense frame-based data while permitting necessary activation increases on sparse event-based data.

B. Learnable Neuron Dynamics

To enhance temporal feature extraction, the standard Leaky Integrate-and-Fire (LIF) neuron parameters are made learnable:

• Parameters: The membrane decay rate ($\beta$) and firing threshold ($V_{thr}$) are optimized during training.
• Efficiency: These are implemented as layer-wise scalars, adding negligible parameter overhead ($O(L)$) while allowing the network to adapt its temporal integration window to specific input statistics.

C. Continual Learning Protocol

• Experience Replay: A class-balanced episodic memory buffer stores samples from previous tasks to mitigate forgetting.
• Class-Incremental Learning (Class-IL): The model learns disjoint subsets of classes sequentially without access to task identifiers during inference.

3. Key Contributions

1. Energy-Aware Framework: Introduction of a proportional control-based spike scheduler that treats energy (spike rate) as a first-class optimization objective alongside accuracy.
2. Discovery of Modality-Dependent Duality: The paper formalizes a fundamental duality in SNN continual learning:
   • Frame-based Data: Spike budgeting acts as a sparsity-inducing regularizer, pruning redundant Poisson activations to improve generalization.
   • Event-based Data: Spike budgeting acts as a controlled activation booster, relaxing constraints to prevent underfitting and enable temporal feature extraction.
3. Comprehensive Cross-Modality Evaluation: A systematic study across five benchmarks (MNIST, CIFAR-10, N-MNIST, CIFAR-10-DVS, DVS-Gesture) demonstrating consistent improvements in the accuracy-energy trade-off.

4. Experimental Results

The framework was evaluated against a baseline of Experience Replay (C1) and a naive sequential baseline (C0).

• Frame-Based Datasets (MNIST, CIFAR-10):

  • MNIST: Achieved a 2.31% accuracy increase (93.44% $\to$ 95.75%) while reducing spike rates by 47% (15.31% $\to$ 8.07%).
  • CIFAR-10: Improved accuracy by 1.76% while reducing spikes by 17.6%.
  • Interpretation: The scheduler successfully pruned redundant activations induced by Poisson encoding, acting as an effective regularizer.

• Event-Based Datasets (N-MNIST, CIFAR-10-DVS, DVS-Gesture):

  • DVS-Gesture: Achieved a massive 17.45 percentage point accuracy gain (74.48% $\to$ 91.93%) with only a 0.24% absolute increase in spike rate (0.48% $\to$ 0.72%).
  • Interpretation: Relaxing the budget allowed the network to capture complex temporal dynamics in gesture sequences without violating neuromorphic sparsity constraints (staying well below 1% spike rate).

• Ablation Studies:

  • Experience Replay (C1): Essential for eliminating catastrophic forgetting (improving accuracy from ~19% to ~93% on MNIST).
  • Learnable Dynamics (C2): Provided modest accuracy gains without energy cost.
  • Scheduler Alone (C3): Reduced spikes but slightly hurt accuracy.
  • Combined (C4): Synergistic effect yielded the best results, balancing regularization and plasticity.

5. Significance and Impact

• Hardware Viability: The method directly addresses the constraints of neuromorphic hardware (e.g., Intel Loihi, IBM TrueNorth), where power is dominated by synaptic events. By reducing spike rates on frame data and maintaining ultra-low rates on event data, the approach ensures deployment feasibility.
• Paradigm Shift: It challenges the assumption that energy efficiency and accuracy are always in conflict. On frame-based data, the paper demonstrates a Pareto improvement (higher accuracy, lower energy).
• Adaptive Intelligence: The framework proves that "one-size-fits-all" sparsity constraints are suboptimal. Instead, energy constraints must be data-aware, adapting dynamically to the input modality to optimize the plasticity-stability trade-off.
• Future Directions: The work paves the way for software-defined activity constraints in neuromorphic systems, enabling heterogeneous sensor suites (mixing cameras and event sensors) to operate under a unified, energy-efficient continual learning policy.