SpikeCLR: Contrastive Self-Supervised Learning for Few-Shot Event-Based Vision using Spiking Neural Networks

1. Problem Statement

Event-based vision sensors (Dynamic Vision Sensors or DVS) offer significant advantages over traditional frame-based cameras, including microsecond temporal resolution, high dynamic range, and low power consumption. When paired with Spiking Neural Networks (SNNs), they enable energy-efficient processing on neuromorphic hardware.

However, the deployment of SNNs for event-based vision is severely hindered by data scarcity:

• Label Scarcity: Annotating event data is costly and time-consuming. Existing benchmarks are small compared to frame-based datasets.
• Synthetic Bias: Many datasets are created by re-recording static images with moving cameras, failing to capture naturally dynamic scenes.
• Few-Shot Limitations: In real-world applications (e.g., adapting to new sensors or domains), only a handful of labeled examples ($k \in \{1, 10, 20, 50\}$) are often available per class. Standard supervised learning performs poorly in these low-data regimes.

The core challenge is to learn robust, transferable visual representations from unlabeled event data to enable effective few-shot and semi-supervised learning, while addressing the unique constraints of SNNs (non-differentiable spike operations).

2. Methodology: SpikeCLR

The authors propose SpikeCLR, a contrastive self-supervised learning (SSL) framework adapted for SNNs and event data.

A. Architecture and Training

• Backbone: The framework utilizes SEW-ResNet-18 (Spiking Element-Wise ResNet-18) as the primary encoder. It also evaluates lightweight variants (separable convolutions) and Spiking VGG9 to ensure architecture agnosticism.
• Neuron Model: Leaky Integrate-and-Fire (LIF) neurons are used.
• Training Mechanism: Since spike generation is non-differentiable, the authors employ Surrogate Gradient training. They approximate the gradient of the Heaviside step function with a smooth function (e.g., arctan derivative) to allow backpropagation through time (BPTT).
• Input Representation: Event streams are converted into normalized event histograms (spatio-temporal tensors with two channels for positive/negative polarities) to leverage standard convolutional operations.

B. Contrastive Objective (NT-Xent)

The framework adapts the SimCLR approach:

1. Augmentation: Two stochastic views ($x_1, x_2$) are generated from a single event recording.
2. Encoding: Both views pass through the SNN encoder $f(\cdot)$ to produce features, followed by a spiking projection head $g(\cdot)$.
3. Temporal Aggregation: Since SNNs produce a sequence of embeddings over time steps $T$, the embeddings are time-averaged to create a single vector per view ($\bar{z}$).
4. Loss Function: The NT-Xent (Normalized Temperature-scaled Cross Entropy) loss is applied. It maximizes agreement between the two augmented views of the same sample (positive pair) while minimizing agreement with views from other samples in the batch (negative pairs).

C. Event-Specific Data Augmentations

A critical contribution is the design of augmentations tailored to the sparse, asynchronous, and polarity-based nature of event data. The authors define three families:

1. Spatial Transformations: Random resized cropping, horizontal flipping, and rolling (cyclic shifting). These preserve object identity while altering spatial support.
2. Polarity/Intensity Transformations: Global scaling (brightness-like) and mean shifting (contrast-like) applied to the accumulated event values. This simulates sensor gain/threshold changes without altering spatial structure.
3. Temporal Transformations: Random temporal cropping (sampling a specific time window from the event stream). This forces the model to learn invariance to motion phase and event timing.

3. Key Contributions

1. First Contrastive SSL for SNNs in Few-Shot Regimes: SpikeCLR is the first framework to systematically evaluate contrastive self-supervised learning for SNNs specifically under label-scarce (few-shot) conditions.
2. Surrogate Gradient Adaptation: Successfully adapts SimCLR-style contrastive learning to the non-differentiable domain of SNNs using surrogate gradients.
3. Event-Specific Augmentation Suite: Introduces a novel set of spatial, temporal, and polarity augmentations that are proven to be more effective for event data than standard frame-based augmentations.
4. Comprehensive Evaluation: Extensive experiments across four major benchmarks (CIFAR10-DVS, N-Caltech101, N-MNIST, DVS-Gesture) covering both static-image-derived and dynamic event-stream datasets.

4. Experimental Results

The framework was evaluated using Linear Probing (frozen backbone, trained classifier) and Fine-Tuning (end-to-end training) protocols.

• Performance in Low-Data Regimes:

  • On CIFAR10-DVS, SpikeCLR significantly outperforms supervised baselines trained from scratch in few-shot scenarios (1-shot to 50-shot).
  • In the 1-shot setting, SpikeCLR achieved 20.2% accuracy vs. 11.9% for supervised learning.
  • In the 9-shot setting (approx. 1% of data), it reached 35.2% vs. 26.0%.
  • Similar gains were observed on N-Caltech101, N-MNIST, and DVS-Gesture.

• Ablation Studies:

  • Augmentation Importance: Temporal and Polarity augmentations were found to be more critical than spatial ones for event data. Combining Spatial + Temporal yielded the best 1-shot performance (20.3%). The full pipeline (Spatial + Temporal + Polarity) achieved the highest overall accuracy.
  • Loss Computation: Computing the contrastive loss at each timestep independently (Temporal NT-Xent) provided a stronger learning signal (+1.0% to +2.1% improvement) compared to averaging embeddings before computing the loss.
  • Transfer Learning: Models pretrained on one dataset (e.g., CIFAR10-DVS) successfully transferred to another (e.g., DVS-Gesture), outperforming training from scratch in few-shot settings. This confirms the learned features capture dataset-agnostic spatio-temporal primitives.
  • Data Scaling: Increasing the amount of unlabeled pretraining data (from 10K to 60K samples on N-MNIST) consistently improved downstream performance, demonstrating the scalability of the approach.

5. Significance and Impact

• Bridging the Data Gap: SpikeCLR addresses the primary bottleneck in event-based vision—the lack of labeled data—by enabling effective learning from unlabeled streams.
• Energy Efficiency: By utilizing SNNs and demonstrating that high performance can be achieved with minimal labeled data (reducing the need for expensive annotation and full retraining), the work promotes the viability of event-based systems on resource-constrained neuromorphic hardware.
• Generalizability: The framework proves that self-supervised pretraining creates robust representations that transfer across different sensors, recording protocols, and dynamic vs. static scenarios.
• Future Directions: The paper highlights that while current contrastive methods require large batch sizes (a memory bottleneck for SNNs), future work could explore non-contrastive methods (like SimSiam or BYOL) to further optimize for embedded deployment.

In summary, SpikeCLR establishes a new state-of-the-art for training Spiking Neural Networks on event data, proving that self-supervised learning with tailored augmentations is the key to unlocking the potential of event-based vision in label-scarce environments.