💻 computer science

Sequence-SOD: Bio-inspired Sequence-aware Spiking ObjectDetection for Event Cameras

The paper introduces Sequence-SOD, a bio-inspired Spiking Neural Network detector that processes continuous event sequences while preserving membrane potentials across time steps, thereby significantly improving detection accuracy on event camera data compared to traditional single-interval approaches.

Original authors: Katharina Bendig, René Schuster, Didier Stricker

Published 2026-07-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Katharina Bendig, René Schuster, Didier Stricker

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a movie by only looking at a single, frozen photograph every few seconds. You might miss the speed of a car, the direction a bird is flying, or the subtle shift in a character's expression. This is how standard cameras work for computers: they take a "snapshot" of the world, process it, and then forget everything before taking the next snapshot. But in the real world, things don't stop and start; they flow. To fix this, scientists invented "event cameras." Instead of taking pictures, these cameras act like a super-fast, super-sensitive eye. They only report when something changes—like a pixel noticing a car moving or a light flickering. This creates a stream of tiny, asynchronous signals called "events" rather than heavy, static images.

To make sense of this rapid-fire stream of changes, researchers use "Spiking Neural Networks" (SNNs). Think of an SNN not as a calculator, but as a digital brain. In a real brain, neurons don't constantly fire; they wait until they have enough information, then they send a quick "spike" of electricity to their neighbors. This makes them incredibly energy-efficient because they only do work when necessary. The big question in this field is: How do we teach these digital brains to understand a continuous movie of events without forgetting what happened a split second ago? If the brain forgets the past every time it makes a guess, it can't track a fast-moving car or predict where a pedestrian will step next.

This paper introduces a clever new way to train these digital brains, called Sequence-SOD. The researchers noticed that previous methods were treating the event stream like a series of isolated snapshots. Every time the computer made a guess about what it saw, it would wipe its memory clean and start over. It was like trying to solve a mystery by looking at one clue, writing down a guess, and then immediately erasing your brain before looking at the next clue. The authors argue that this wastes the most valuable part of event data: the timing.

Instead, Sequence-SOD teaches the network to look at a whole "sequence" of events at once. Imagine you are watching a magic trick. If you only see the magician's hand at the very end, you might be confused. But if you watch the whole sequence—the setup, the misdirection, and the reveal—you understand the trick. Similarly, this new method feeds the network a continuous stream of events (specifically, chunks of 125 milliseconds) and keeps the network's internal "memory" (called membrane potentials) active throughout the whole sequence. The network doesn't reset its state between the different moments in that sequence; it lets the information from the first few milliseconds flow into the next, just like a real brain does.

The results of this approach are quite promising. When the researchers tested their method on a dataset of driving scenes (the Gen1 Automotive Detection Dataset), they found that keeping the memory active made a real difference. Without any fancy extra tricks, training on these longer sequences improved the system's ability to detect cars and pedestrians. The accuracy score, known as mAP, jumped from 23.38 (for the old "reset-every-time" method) to 25.30. When they added some standard data tricks (like flipping the images or zooming in) to the training, the score climbed even higher to 26.88.

The paper also highlights that this method is incredibly efficient. Because the network only "spikes" (does work) when it needs to, it uses far less energy than traditional computer vision systems. Theoretically, this setup allows the system to make a new prediction every 25 milliseconds, which is a frequency of 40 Hz. This means the digital brain can track fast-moving objects in real-time without getting tired or running out of battery power.

The authors are careful to note that they didn't invent a brand-new type of brain architecture; they used a known design (a Spiking DenseNet) and simply changed how they trained it. They argue that the key to better performance isn't just building bigger or more complex networks, but teaching them to respect the flow of time. By treating the event stream as a continuous story rather than a pile of disconnected pages, Sequence-SOD helps these energy-efficient digital brains finally start to "see" the world the way it actually moves.

Technical Summary: Sequence-SOD

Problem Statement

Event cameras offer high temporal resolution, wide dynamic range, and low power consumption by asynchronously reporting local intensity changes, making them ideal for autonomous driving. However, their sparse and asynchronous nature renders traditional frame-based object detection methods ineffective. While Spiking Neural Networks (SNNs) are a natural fit for processing such data due to their brain-inspired, stateful dynamics, current SNN-based object detectors suffer from a critical limitation: they typically process isolated event intervals and reset the network's internal state (membrane potentials) after each prediction. This approach fails to leverage the continuous temporal information inherent in event streams, treating perception as a series of independent snapshots rather than a continuous process. Consequently, these models underutilize temporal cues that could improve detection accuracy and robustness.

Methodology

The authors introduce Sequence-SOD, a sequence-aware SNN object detector designed to process extended event sequences while preserving the internal state of the network across time intervals.

Data Processing and Sequence Construction:

Event Accumulation: Asynchronous event streams are accumulated into short time intervals (125 ms).
Discretization: Each interval is further divided into $T=5$ discrete time steps (25 ms each) to facilitate GPU-based simulation.
Sequence Formation: During training, the model processes sequences of length $s$ (where $s$ represents the number of consecutive 125 ms intervals). A sequence may contain multiple bounding box labels at different time points, interspersed with unlabeled intervals.
State Management: Crucially, the membrane potentials of the spiking neurons are not reset between intervals within a single sequence. The state is only reset between independent sequences. This allows the network to retain a "memory" of recent events through leaky integration and decay, enabling it to prioritize significant temporal cues and discard outdated information.

Network Architecture:

The model utilizes an SSD-style detection framework with a Spiking DenseNet backbone (depth 121, growth rate 24).
Standard ReLU activations are replaced with Parametric Leaky Integrate-and-Fire (PLIF) neurons, where the decay rate is a trainable parameter, allowing neurons to adapt their temporal retention to the event stream statistics.
Batch normalization is used during training but mathematically absorbed into convolutional layers for inference, ensuring the final model remains a pure SNN.
Prediction Strategy: For a given interval, the network output is accumulated over the $T$ internal time steps and averaged to produce a single bounding box and class prediction. This prevents the same label from being treated as five independent training targets.
Training: The model is trained using surrogate gradients to handle the non-differentiable spike function. The detection loss is applied only to labeled intervals, while the network state propagates through unlabeled intervals.

Key Contributions

Sequence-Aware Training Strategy: The paper proposes a training and evaluation paradigm that treats membrane potentials as the model's temporal memory. Unlike prior methods that reset states after every prediction, Sequence-SOD maintains state continuity across multiple labeled and unlabeled intervals within a sequence.
Performance Improvement via Temporal Continuity: The authors demonstrate that training on extended sequences (e.g., length $s=5$ ) significantly outperforms single-interval training ( $s=1$ ) on the Gen1 Automotive Detection Dataset, even when the test sequence length is short.
Energy Efficiency Analysis: The study evaluates the theoretical energy consumption of the SNN architecture compared to an equivalent Artificial Neural Network (ANN), highlighting the reduction in operations due to sparse spiking activity.

Results

Experiments were conducted on the Gen1 Automotive Detection Dataset (39 hours of automotive event data).

Accuracy (mAP):
- Single-interval training (baseline) achieved an mAP of 23.38.
- Sequence-aware training (sequence length 5) without augmentation improved mAP to 25.30.
- With event-data augmentation (horizontal flipping, zooming in/out), the mAP further increased to 26.88.
- The improvement is attributed to the network's ability to learn a more meaningful inner state representation that prioritizes essential temporal information.
Temporal Resolution: The model processes 25 ms time steps, theoretically enabling a prediction frequency of 40 Hz.
Energy Efficiency: By replacing Multiply-Accumulate (MAC) operations with Accumulate (AC) operations and leveraging sparse activations, the SNN demonstrates a significant reduction in estimated energy consumption compared to an equivalent ANN, particularly when deployed on neuromorphic hardware.
Ablation: Longer training sequences forced the network to learn better state representations. While performance slightly decreased when testing a model trained on short sequences against full sequences, the model trained on longer sequences maintained robust performance across all test conditions.

Significance and Claims

The paper claims that Sequence-SOD highlights sequence-aware training as a complementary direction to architectural improvements for event-based SNN detection. The authors argue that the primary bottleneck in current SNN detectors is not necessarily the architecture itself, but the failure to exploit the continuous nature of event streams by resetting internal states prematurely.

By preserving membrane potentials across intervals, the model mimics the stateful nature of biological perception, allowing it to integrate evidence over time. The authors modestly state that while their method improves detection performance and maintains the energy-efficiency benefits of sparse spiking computation, it does not fully close the accuracy gap with traditional ANNs or hybrid architectures. They emphasize that their contribution isolates the effect of temporal state preservation within a standard SSD-style Spiking DenseNet, proving that sequence-aware training alone yields measurable gains without requiring complex architectural overhauls.

The work suggests that for real-world applications where detectors operate on continuous streams, maintaining a temporal memory via membrane potentials is essential for leveraging the full potential of event-based vision.