💻 computer science

Spiking Patches: Asynchronous, Sparse, and Efficient Tokens for Event Cameras

The paper introduces "Spiking Patches," an asynchronous and sparse tokenization method for event cameras that preserves their unique properties while achieving faster inference and comparable or superior accuracy to existing frame and voxel-based approaches across various vision tasks.

Original authors: Christoffer Koo Øhrstrøm, Ronja Güldenring, Lazaros Nalpantidis

Published 2026-09-02

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Christoffer Koo Øhrstrøm, Ronja Güldenring, Lazaros Nalpantidis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots and self-driving cars have long relied on standard cameras that capture the world like a movie camera, taking a complete picture of a scene every fraction of a second. This method works well for many things, but it creates a heavy burden of data, much of which is just empty space or unchanged background. A newer type of sensor, known as an event camera, operates on a different principle. Instead of taking full pictures, it only records changes. When a pixel on the sensor notices a shift in brightness, it sends a single signal, or event, at that exact moment. This means the camera produces a stream of isolated signals that are asynchronous, happening whenever something changes, rather than in a fixed rhythm. It also means the data is sparse, containing only the information about where movement or change occurred, leaving the rest of the scene silent. While this approach is incredibly efficient and fast, it has been difficult to feed this unique stream of data into the artificial intelligence models that robots use to understand their surroundings.

For years, researchers have tried to solve this by forcing these asynchronous signals into the old, familiar format of standard images. They would group the events into fixed time windows to create a frame, much like stitching together a series of snapshots. However, this process destroys the very qualities that make event cameras special. By waiting for a fixed time to pass before creating an image, the system loses the ability to react instantly to sudden changes, and by filling in the empty spaces, it loses the efficiency of having only the relevant data. A team of researchers at the Technical University of Denmark has now proposed a different way forward. They developed a new method called Spiking Patches, which treats groups of these changing signals as single units of information without forcing them into a rigid, time-based grid. This approach allows the data to remain asynchronous and sparse, preserving the speed and efficiency of the original sensor while making it compatible with powerful modern AI models.

The researchers designed their system by looking at how biological neurons work. In the brain, a neuron waits until it receives enough signals to reach a certain threshold before it fires a spike. The team applied this same logic to the event camera data. They divided the camera's view into a grid of small squares, or patches. As events arrive, they accumulate within each patch, raising a virtual potential. Once the number of events in a patch reaches a specific limit, the patch "fires," creating a token that represents that group of events. This token is then sent to the AI model for processing. Crucially, this firing happens independently for each patch and at the exact moment the threshold is reached, rather than waiting for a clock to tick. This means the system can react to a fast-moving object the instant enough events accumulate, without waiting for a full frame to be built.

To test if this idea worked in practice, the team compared their Spiking Patches against the two most common ways of handling event data: standard frames and 3D blocks of data called voxels. They tested the method on three different types of AI architectures, including networks designed for graphs, point clouds, and transformers, using tasks like recognizing hand gestures and detecting cars in driving footage. The results were striking. In terms of speed, the new method was significantly faster than the alternatives. When recognizing gestures, the Spiking Patches system was up to 3.4 times faster than the voxel method and up to 10.4 times faster than the frame-based method. This speedup came without sacrificing accuracy; in many cases, the new method was just as accurate as the older ones, and in some instances, it was even more accurate, improving gesture recognition by up to 3.8 percentage points and object detection by up to 1.4 percentage points.

The study also confirmed that the method successfully kept the data sparse and asynchronous. The researchers measured how quickly the system could gather enough information to react to a scene compared to the raw stream of events. They found that the Spiking Patches system could react almost as quickly as the raw events themselves, with only a tiny delay of a few milliseconds, whereas the frame-based methods were forced to wait for a fixed time interval, introducing a much longer lag. Furthermore, the new method reduced the amount of data the computer had to process by a factor of at least 1.5 compared to frames and voxels, and in some cases, by hundreds of times compared to the raw event stream. This reduction in data size is what allows the system to run so much faster, as the computer has fewer items to analyze.

One of the most practical aspects of this discovery is that the system is fast enough to handle real-time applications. The researchers implemented the tokenization process in a programming language known for its speed and found that the system could generate tokens faster than new events were arriving at the camera. This means the system can keep up with the flow of information in real-world scenarios, such as a car driving down a highway, without falling behind. The team also explored how different settings affected the performance, finding that they could adjust the sensitivity of the system to trade off between the number of tokens generated and the precision of the result. For example, by introducing a brief pause after a patch fires, they could reduce the number of tokens by four times while keeping the accuracy nearly the same.

While the results are promising, the researchers note that the method requires careful tuning of the threshold that triggers a token. If the threshold is set too low, the system might fire too often, creating too much data. If it is set too high, it might miss important details. The team suggests that future work could focus on making this threshold adapt automatically to the scene. They also plan to test the method on other types of tasks, such as tracking moving objects or calculating the flow of motion in a scene. For now, however, the work demonstrates that it is possible to bridge the gap between the unique, asynchronous nature of event cameras and the powerful AI models that drive modern robotics. By treating groups of events as single, meaningful units, the researchers have shown that we can keep the speed and efficiency of event cameras without losing the ability to use the most advanced tools in artificial intelligence.

Technical Summary: Spiking Patches

Problem Statement

Event cameras offer significant advantages for robotic applications due to their asynchronous and spatially sparse output, which allows for rapid reaction to sudden changes and reduced computational load. However, existing deep learning perception approaches typically convert these asynchronous event streams into synchronous, dense frame-based representations or synchronous voxels. This conversion discards the inherent asynchrony and spatial sparsity of the data. While some methods attempt to reclaim these properties post-conversion, the paper argues that it is more beneficial to utilize a representation that preserves these properties by default, ensuring that any model using the representation inherently maintains them. Currently, no representation exists that successfully tokenizes events while preserving both asynchrony and spatial sparsity without sacrificing accuracy.

Methodology: Spiking Patches

The authors propose Spiking Patches, a spatio-temporal tokenizer designed specifically for event cameras. The method is built upon two key observations:

Individual events contain little information, but groups of events (patches) contain meaningful information, similar to how pixel patches function in standard computer vision.
Events exist in a spatio-temporal space and must be grouped across both dimensions.

Core Mechanism:

Spatial Grouping: The camera viewport is divided into a grid of non-overlapping patches of size $P \times P$ . Events are restricted to their respective patches to ensure spatial proximity.
Temporal Grouping (Spiking): Each patch is treated as a biological spiking neuron. The patch maintains a potential $u$ , which increases by a constant value ( $I=1$ ) for every event arriving within the patch.
Thresholding: When the patch potential exceeds a threshold $\sigma$ , a "spike" occurs. A token is generated consisting of all events accumulated since the last spike. The potential is then reset to the resting potential ( $u_{rest}=0$ ).
Refractory Period: To control temporal resolution and reduce token count, a refractory period $T$ can be applied. During this period following a spike, incoming events are ignored and do not contribute to the potential or token generation.

This mechanism ensures that tokens are generated only when sufficient activity occurs, preserving spatial sparsity, and that generation is tied directly to event arrival times, preserving asynchrony. The tokenizer operates with a time complexity of $O(n)$ , where $n$ is the number of events.

Key Contributions

Spiking Patches Tokenizer: The introduction of a new event representation that tokenizes events into spatio-temporal groups while preserving the unique asynchronous and sparse nature of event cameras.
Performance Gains: Demonstration that Spiking Patches matches or surpasses the accuracy of frame-based and voxel-based representations while significantly improving inference speed.
Property Preservation: Validation that the method maintains levels of asynchrony and spatial sparsity comparable to raw event streams.
Efficiency: Evidence that the tokenizer reduces input sizes by at least $1.5\times$ compared to frames and voxels, and that the implementation (in Rust) generates tokens faster than events arrive, making it suitable for real-time applications.

Experimental Results

The authors evaluated Spiking Patches using Graph Neural Networks (GNNs), Point Cloud Networks (PCNs), and Transformers on gesture recognition (DvsGesture, SL-Animals-DVS) and object detection (GEN1).

Accuracy:
- Gesture Recognition: Spiking Patches matched the accuracy of voxels for GNNs and PCNs. For Transformers, it surpassed frames by 1.1% and voxels by 0.4%. On SL-Animals-DVS, it achieved a 3.8% absolute improvement over voxels for GNNs.
- Object Detection: Spiking Patches achieved results comparable to frames (within 0.1 mAP) and outperformed voxels, with mAP improvements of 1.4% (GNN), 0.4% (PCN), and 1.3% (Transformer).
- Event Baseline: Compared to using raw events directly (subsampled), Spiking Patches improved accuracy (up to 4.1% for PCNs) while significantly reducing inference time.
Speed:
- Inference times were up to 3.4 $\times$ faster than voxel-based tokens and up to 10.4 $\times$ faster than frame-based tokens.
- Tokenization speed on an Intel I9 CPU reached 39.8M events/second, nearly double the speed of voxelization (23.0M events/second) and far exceeding the incoming event rate of the GEN1 dataset.
Efficiency & Sparsity:
- Input sizes were reduced by at least $1.5\times$ compared to voxels and significantly more compared to frames (up to $1120\times$ reduction in token count compared to raw events).
- Spatial sparsity analysis showed that with appropriate thresholds ( $\sigma \ge 250$ ) or refractory periods ( $T \ge 100$ ms), Spiking Patches maintains sparsity levels comparable to raw events, unlike frames which are dense.
Asynchrony:
- Analysis of event accumulation showed that Spiking Patches reacts asynchronously, following the slope of the event curve with a minimal delay (5–20 ms depending on $\sigma$ ), allowing for faster reaction times than synchronous methods.

Significance and Claims

The paper claims that Spiking Patches represents a novel direction for event camera representation by successfully bridging the gap between the raw asynchronous nature of events and the requirements of modern deep learning architectures (GNNs, PCNs, Transformers).

The authors assert that their method allows models to preserve the unique properties of event cameras (asynchrony and spatial sparsity) by default, rather than attempting to recover them after conversion. The results suggest that this approach does not come at the cost of accuracy; in fact, it often improves it while delivering substantial speedups. The authors conclude that Spiking Patches is efficient, compatible with existing architectures, and highly relevant for real-time robotic applications, encouraging further research into tokenization strategies for event-based vision.

Limitations noted by the authors:

The current formulation requires careful tuning of the spike threshold $\sigma$ .
Future work is needed to explore adaptive thresholding and to validate the method on additional network types and tasks (e.g., optical flow, object tracking).