⚡ electrical engineering

CATFormer: When Continual Learning Meets Spiking Transformers With Dynamic Thresholds

CATFormer is a scalable spiking transformer framework that prevents catastrophic forgetting in class-incremental learning by employing a Dynamic Threshold Leaky Integrate-and-Fire neuron model with context-adaptive thresholds and a Gated Dynamic Head Selection mechanism, achieving superior performance on both static and neuromorphic datasets without rehearsal.

Original authors: Vaishnavi Nagabhushana, Kartikay Agrawal, Ayon Borthakur

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vaishnavi Nagabhushana, Kartikay Agrawal, Ayon Borthakur

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Goldfish" Robot

Imagine you are teaching a robot to recognize different animals. First, you show it pictures of cats. It learns perfectly. Then, you show it pictures of dogs. When you ask it to identify a cat again, it forgets everything it knew about cats and only knows dogs. This is called Catastrophic Forgetting.

Standard AI models are like goldfish; they have a very short memory. Every time they learn something new, they wipe the slate clean, overwriting the old information. In the real world, robots and smart devices can't afford to forget. They need to learn new skills over years without needing a massive hard drive to store every single photo they've ever seen (which is often impossible due to privacy laws or battery limits).

The Brain's Secret: The "Volume Knob"

The authors of this paper looked at how the human brain solves this. The brain doesn't just store memories in a static file; it uses neuromodulators (chemicals like dopamine or acetylcholine) to change how neurons behave.

Think of a neuron like a light switch.

Standard AI: To learn a new task, the AI tries to rewire the entire house (changing all the light switches). This accidentally breaks the lights in the rooms you were already using.
The Brain (and CATFormer): Instead of rewiring the house, the brain just turns the sensitivity of the switches up or down. It says, "For this specific task, be very sensitive to cats. For that task, be less sensitive."

The Solution: CATFormer

The researchers built a new AI called CATFormer (Context Adaptive Threshold Transformer). It's a "Spiking Transformer," which is a fancy way of saying it's a super-efficient, brain-like computer that uses electrical spikes instead of heavy math.

Here is how it works, broken down into three simple parts:

1. The "Dynamic Threshold" (The Smart Volume Knob)

In CATFormer, the neurons have a "firing threshold." Think of this as a volume knob.

Old way: To learn a new task, the AI changes the wiring (the weights).
CATFormer way: The main wiring stays frozen (it remembers the past). When a new task arrives, the AI simply turns the volume knob for specific neurons.
Analogy: Imagine a library. The books (knowledge) stay on the shelves. When you want to study "History," you don't rewrite the books; you just put a "History" bookmark on the specific pages you need. When you switch to "Science," you move the bookmark. The books remain safe and unchanged.

2. The "Gated Head" (The Bouncer)

The system uses a "Gating Mechanism." Think of this as a bouncer at a club.

When you walk in with a new input (a picture), the bouncer checks your ID.
"Oh, you're a picture of a bird? Go to the Bird Room."
"Oh, you're a picture of a car? Go to the Car Room."
Each "Room" has its own specific volume knobs (thresholds) tuned perfectly for that topic. This ensures the robot doesn't get confused between a cat and a car.

3. The "No-Notes" Policy (Rehearsal-Free)

Most AI systems need to keep a "rehearsal buffer"—a folder of old photos they look at while learning new things to prevent forgetting.

CATFormer doesn't need this folder. Because it uses the "Volume Knob" method, it never overwrites old knowledge. It learns new things without needing to look back at old data. This saves massive amounts of memory and protects privacy.

The Amazing Result: "Reverse Forgetting"

Usually, as AI learns more tasks, it gets worse at the old ones.

Normal AI: Learns 10 tasks = 90% accuracy. Learns 50 tasks = 50% accuracy. (It forgets).
CATFormer: Learns 10 tasks = 68% accuracy. Learns 50 tasks = 75% accuracy.

The authors call this "Reverse Forgetting." It's like a musician who, after learning 50 new songs, actually gets better at playing the first song because their brain has become so good at organizing the information. The more tasks they learn, the smarter the "Volume Knob" system becomes at sorting them.

Why This Matters

This is a huge deal for physical AI (robots, self-driving cars, smart sensors) because:

Battery Friendly: It uses "Spiking" (like a brain), which is much more energy-efficient than standard computers.
Memory Saver: It doesn't need to store thousands of old photos, making it perfect for small devices.
Real-World Ready: It can learn continuously over months or years without crashing or forgetting, just like a human does.

In a nutshell: CATFormer is a robot brain that learns new skills by simply adjusting its "sensitivity settings" rather than rewriting its entire memory, allowing it to learn forever without forgetting a thing.

1. Problem Statement

The paper addresses the challenge of Catastrophic Forgetting in Deep Neural Networks (DNNs) within the context of Class-Incremental Learning (CIL).

The Core Issue: Standard DNNs (MLPs, CNNs, Transformers) fail in real-world scenarios where data arrives sequentially. When trained on new tasks, they overwrite previously acquired knowledge, leading to a sharp drop in performance on old tasks.
Constraints of Physical AI: In resource-constrained environments (robotics, edge devices, battery-operated agents), data rehearsal (storing and replaying past data) is often infeasible due to memory limits, energy constraints, and privacy regulations.
Limitations of Existing SNNs: While Spiking Neural Networks (SNNs) offer energy efficiency, existing SNN-based CIL methods (mostly CNN-based) suffer from performance degradation as the number of tasks increases. Furthermore, the potential of Spiking Transformers for continual learning remains largely unexplored.

2. Methodology: CATFormer

The authors propose CATFormer (Context Adaptive Threshold Transformer), a scalable framework designed for rehearsal-free continual learning in Spiking Vision Transformers. The architecture is inspired by biological neuromodulation (e.g., acetylcholine), which regulates neuronal excitability to prevent interference.

Key Architectural Components:

Dynamic Threshold Leaky Integrate-and-Fire (DTLIF) Neuron:
- Unlike standard LIF neurons with fixed thresholds, CATFormer employs context-adaptive, learnable thresholds ( $\phi^{(k)}$ ).
- Mechanism: During the training of task $k$ , the synaptic weights of the backbone (SpikFormer) are frozen. Only the task-specific thresholds and the new classification head are updated via gradient descent.
- Biological Analogy: This mimics neuromodulators that alter membrane excitability, allowing the network to encode new memories without overwriting old ones by changing how neurons fire rather than what they connect to.
Gated Dynamic Head Selection (G-DHS):
- A lightweight Gating MLP is trained to predict the current task ID based on input features.
- Inference Workflow:
  1. The model extracts base features using a standard threshold.
  2. The Gating MLP predicts the task ID ( $k^*$ ).
  3. The system switches to the specific dynamic threshold $\phi^{(k^*)}$ and the corresponding task-specific classification head $W_{k^*}$ for final classification.
Two-Stage Training Protocol:
- Stage 1 (Task 0): Jointly optimize the backbone, initial thresholds, and the first head.
- Stage 2 (Subsequent Tasks $k > 0$ ): Freeze the backbone parameters. Initialize a new head and task-specific thresholds. Optimize only the new head and thresholds.
- No Rehearsal: The system does not store raw data or exemplar buffers from previous tasks.

3. Key Contributions

First Spiking Transformer for CIL: Introduces the first Class-Incremental Learning framework specifically designed for Spiking Vision Transformers, bridging the gap between Transformer capabilities and SNN energy efficiency.
Novel Plasticity Mechanism: Proposes a biologically inspired mechanism where dynamic thresholds serve as the primary vehicle for plasticity, while the backbone remains static. This eliminates the need for network growth or data replay.
Scalability & "Reverse Forgetting": Demonstrates that the model does not just maintain accuracy but actually improves as the number of tasks increases (up to 100 tasks), a phenomenon the authors term "Reverse Forgetting."
Hardware Efficiency: The approach is highly parameter-efficient. Only ~~1.4M parameters are updated per new task (mostly thresholds and heads), and the memory overhead for storing thresholds is negligible (~~64 KB for 50 tasks).

4. Experimental Results

The model was evaluated on static datasets (CIFAR-10/100, Tiny-ImageNet) and neuromorphic datasets (CIFAR10-DVS, SHD).

Performance on CIFAR-100 (50 Tasks):
- CATFormer achieved 75.66% accuracy on 50 tasks.
- In contrast, state-of-the-art baselines like DSD-SNN dropped from ~60% to ~50%, and regularization methods (EWC, SI) dropped to <10%.
- Reverse Forgetting Trend: Accuracy increased from 68.33% (10 tasks) to 75.66% (50 tasks).
Performance on Tiny-ImageNet (100 Tasks):
- CATFormer achieved 48.56% accuracy on 100 tasks, outperforming non-spiking ANN baselines by 8.45%.
Neuromorphic Datasets:
- On CIFAR10-DVS (2 tasks), CATFormer achieved 83.21% without rehearsal, matching the performance of ALADE-SNN which uses rehearsal buffers.
- On SHD (audio), it achieved 87.85% on 10 tasks, significantly outperforming DSD-SNN.
Ablation Studies:
- Removing dynamic thresholds (Fixed Threshold) caused accuracy to crash to 42.87%, proving the critical role of threshold modulation.
- Removing attention mechanisms (SpikIdentityFormer) reduced performance, confirming the necessity of the Transformer's structured feature extraction.

5. Significance and Impact

Enabling Lifelong Learning on Edge Devices: CATFormer solves the memory and privacy bottlenecks of rehearsal-based methods, making it ideal for autonomous robots and edge AI that must learn continuously over months or years without storing user data.
Energy Efficiency: By leveraging Spiking Transformers and freezing the backbone, the model drastically reduces computational overhead and energy consumption compared to training full networks or using heavy replay buffers.
Biological Plausibility: The approach validates the hypothesis that modulating neuronal excitability (thresholds) is a more effective strategy for preventing interference than synaptic plasticity alone, offering a new direction for brain-inspired AI.
Scalability: The ability to handle 100+ incremental tasks without performance degradation establishes a new benchmark for long-term continual learning in physical AI systems.