CLANE: Continual Learning of Actions on Neuromorphic Hardware from Event Cameras

Technical Summary: CLANE – Continual Learning of Actions on Neuromorphic Hardware from Event Cameras

Problem Statement
The paper addresses the challenge of deploying continual learning (CL) systems for action recognition on edge devices with strict power and memory constraints. Traditional deep learning approaches rely on global error propagation, large activation buffers, and access to past data or replay buffers, which are incompatible with the resource budgets of always-on edge devices. Furthermore, standard visual action recognition struggles with variable lighting, motion blur, and redundant static data. While event cameras offer sparse, asynchronous, and high-dynamic-range data suitable for low-latency processing, no prior system has successfully deployed a full end-to-end continual learning pipeline for event-based action recognition on neuromorphic hardware. Existing neuromorphic CL work has been limited to static spatial inputs (object recognition) or small-scale benchmarks lacking catastrophic forgetting analysis.

Methodology
The authors present CLANE (Continual Learning of Actions on Neuromorphic hardware from Event cameras), an end-to-end system deployed on the Intel Loihi 2 neuromorphic processor. The system integrates three core components:

1. Event-Driven Data Pipeline: The system utilizes the THUE-ACT-50 dataset captured by a CeleX-V event camera. Event streams are binned into fixed temporal windows (e.g., 40ms) to create sparse 2D event-count histograms. These sparse representations are processed directly as spike trains, avoiding the dense tensor operations required by frame-based cameras.
2. Spiking 2D CNN Feature Extractor: A 5-layer spiking convolutional neural network (SNN) extracts spatiotemporal features. The network uses Parametric Leaky Integrate-and-Fire (PLIF) neurons. This feature extractor is pre-trained on 38 base classes using backpropagation through time (BPTT) on a GPU and then frozen for the continual learning phase.
3. CLP-Loihi Module (On-Chip Learning Head): To handle the temporal nature of actions and the hardware constraints of Loihi 2, the authors extend the CLP-SNN algorithm with two novel modules:
   • Temporal Aggregation Layer: Integrates binary spike streams over the duration of an action clip using non-leaky Integrate-and-Fire neurons, outputting "graded spikes" (vectors) that represent the entire action.
   • Normalization Layer: Implements L2-normalization using a fixed-point fast inverse square root algorithm via Loihi 2's microcode engine, as the chip lacks native division instructions.
   • Continual Learning Mechanism: The system employs a self-normalizing, three-factor learning rule. It learns online and class-incrementally by creating new prototype vectors for novel actions when errors occur, without requiring global error propagation or replay buffers.

Key Contributions

• CLP-Loihi: The extension of the CLP-SNN algorithm to spatiotemporal action recognition. This includes the novel implementation of Temporal Aggregation and Fixed-Point Normalization layers directly on Loihi 2 using custom neuron models.
• Full Neuromorphic Deployment: The first end-to-end deployment of a deep spiking CNN feature extractor and a continual learning head on a single Loihi 2 chip. The system processes event-camera inputs without GPU intervention in either the inference or learning loops.
• Rigorous Iso-Algorithm Benchmarking: The paper introduces a cross-platform benchmarking methodology comparing CLANE on Loihi 2 against an Nvidia Jetson Orin Nano. By keeping the learning algorithm (CLP) and model capacity constant across platforms, the authors isolate hardware efficiency gains from algorithmic differences.

Results
Evaluated on the THUE-ACT-50 dataset (50 classes, real-world conditions including lighting variations and occlusions), CLANE demonstrates the following:

• Accuracy: In a 10-shot class-incremental learning task, CLANE achieves 70.4% accuracy. This is competitive with the CLP algorithm running on the GPU (73.0%) and significantly outperforms naive fine-tuning, which suffers from catastrophic forgetting.
• Efficiency: Compared to a sequential CNN+GRU baseline running the same CLP algorithm on the Jetson Orin Nano, CLANE delivers:
  • >100× reduction in energy consumption.
  • 16× lower latency.
  • An Energy-Delay Product (EDP) of 0.018 mJ·s, matching the efficiency of a 3D CNN on the GPU but with significantly lower latency in streaming scenarios.
• Feature Extraction: The spiking CNN on Loihi 2 provides approximately 30× latency reduction and 230× energy reduction compared to the 2D CNN+GRU baseline on the GPU, with only a marginal accuracy cost (2.6%).

Significance and Claims
The paper claims that CLANE establishes neuromorphic hardware as a practical substrate for "always-on, continually adapting perception systems." The authors argue that the architectural alignment between event cameras (sparse sensing), SNNs (sparse computation), and neuromorphic processors (local learning) creates synergistic efficiency gains that are unattainable with von Neumann architectures for sparse, online, power-constrained regimes.

The work highlights that while GPUs excel at dense, batched processing, neuromorphic systems dominate in scenarios requiring low-latency, low-energy, and continuous adaptation without forgetting. The paper concludes that this full-stack approach addresses a critical gap in the field, providing a benchmarking methodology that the neuromorphic continual learning community currently lacks, and demonstrating that large-scale spatiotemporal continual learning is feasible on edge neuromorphic hardware.