💻 computer science

Neural Events: Discrete Asynchronous Autoencoders for Event-Based Vision

This paper proposes a framework that re-tokenizes raw event camera streams into highly compressed, discrete "neural events" using learnable autoencoders, achieving state-of-the-art performance in object detection and classification while reducing data throughput by a factor of two.

Original authors: Roberto Pellerito, Daniel Gehrig, Shintaro Shiba, Davide Scaramuzza

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Roberto Pellerito, Daniel Gehrig, Shintaro Shiba, Davide Scaramuzza

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to watch a movie, but instead of receiving a steady stream of frames, the camera sends you a chaotic, high-speed torrent of individual notes. Each note just says, "Something got brighter here!" or "Something got darker there!" This is how event cameras work. They are incredibly fast and efficient, capturing changes in a scene with microsecond precision. However, for a computer to understand what's happening (like spotting a car or a person), it has to read millions of these tiny, low-value notes every second. It's like trying to understand a conversation by listening to every single syllable spoken by a thousand people at once—it's overwhelming and wasteful.

The paper proposes a clever solution called Neural Events. Think of this as a smart translator that sits between the chaotic camera and the computer brain.

The Problem: Too Much Noise, Too Little Sense

Current methods try to handle this flood of data in two ways, both of which have flaws:

The "Sync" Approach: They try to bundle these notes into fixed-time chunks (like frames in a movie). This loses the super-fast timing information and wastes energy processing empty space.
The "Async" Approach: They try to process every single note as it arrives. While this keeps the timing, the computer gets bogged down trying to build complex maps of connections between notes, which is slow and memory-hungry.

The Solution: The "Smart Summarizer"

The authors created a system that acts like a highly efficient editor for this stream of notes. Here is how it works, using a simple analogy:

1. Dividing the Scene into Neighborhoods
Imagine the camera's view is a giant city. Instead of looking at every single street corner individually, the system divides the city into small neighborhoods (patches).

2. The Local Translator (The Encoder)
Inside each neighborhood, there is a tiny, super-fast translator (a "Discrete Asynchronous Encoder"). As the raw notes (events) pour into a neighborhood, this translator reads them one by one.

Instead of writing down every single note, the translator assigns a secret code to the current situation.
Think of this code like a mood ring or a traffic light. As long as the "mood" of the neighborhood stays the same (e.g., "a car is moving steadily"), the translator keeps the same code. It doesn't need to send a new message for every tiny change.

3. The "Flip" Trigger
Here is the magic trick: The translator only sends a Neural Event (a new message) when the code flips.

If the code changes from "Red" to "Green," that's a signal! It means something significant happened in that neighborhood.
If the code stays "Red" for 1,000 raw notes, the translator ignores the 999 redundant notes and only sends the one "Red" signal.
This is like a security guard who only calls the police when a door actually opens, rather than calling every time a shadow moves across the floor.

4. The Result: A Compressed Stream
The result is a tiny stream of "Neural Events." Each one carries a timestamp, a location, and a rich, high-level code that summarizes what's happening.

Compression: The paper claims this reduces the amount of data by 2 times (halving the traffic) while actually making the information better for the computer to understand.
Efficiency: The system is so efficient that it uses 17 times less computing power than the current best methods to process the same amount of data.

Why This Matters

The authors tested this "Smart Summarizer" on tasks like finding cars and recognizing objects. They found that computers trained on these compressed "Neural Events" performed just as well as, or even better than, computers trained on the raw, messy data or the older, clunky methods.

In short: The paper introduces a way to turn a chaotic flood of raw sensory data into a clean, concise, and highly informative stream of "smart signals." It allows computers to see the world through event cameras without getting overwhelmed by the noise, making it possible to run these powerful systems on smaller, battery-powered devices.

Technical Summary: Neural Events: Discrete Asynchronous Autoencoders for Event-Based Vision

Problem Statement

Event cameras offer exceptional temporal fidelity by capturing scenes as a continuous stream of microsecond-resolution events, where each event signals a localized brightness change. However, individual events carry minimal semantic value, requiring downstream algorithms to integrate massive torrents of low-information data. Current architectures face a fundamental tradeoff:

Synchronous approaches (e.g., CNNs, Transformers) aggregate events into dense feature maps. While effective, they lose the event-driven nature of the data, leading to wasteful processing, storage, and increased latency.
Asynchronous approaches (e.g., Spiking Neural Networks, Graph Neural Networks) preserve sparsity and timing but struggle with scalability. SNNs are limited by slow sequential processing, while GNNs incur massive memory overheads due to dynamic graph construction, often resulting in lower task performance compared to dense methods.

The core challenge is to represent event patterns in a way that is scalable, preserves precise timing and sparsity, and reduces data throughput without sacrificing semantic richness.

Methodology

The paper proposes a framework to re-tokenize raw event streams into a compact set of "neural events." These are semantically rich, spatio-temporally sparse representations that summarize groups of raw events into discrete codes.

1. Neural Event Definition

A neural event $n_i$ is defined by a patch location $p_i$ , a timestamp $\tau_i$ , and a discrete feature $z_i$ selected from a learned codebook $Q$ . The mapping $\phi$ reduces the number of events ( $N' \ll N$ ) and the spatial domain ( $|\Omega'| \ll |\Omega|$ ).

2. Discrete Asynchronous Encoder

The generation of neural events occurs patch-wise and involves four stages:

Input Embedding: Raw events are projected into a continuous vector space using spatial and temporal embeddings (incorporating sine/cosine positional encodings for time).
Linear Transformer (RWKV-7): A linear-attention recurrent model processes the event sequence within each patch. It updates a localized memory state $S_m$ to output a continuous logit vector $o_i$ . This architecture avoids the quadratic complexity of standard Transformers and the sequential bottlenecks of LSTMs.
Quantization: The continuous logits are mapped to a finite set of discrete codes from a learned codebook using a Gumbel Softmax operator (Straight-Through Estimator). This converts analog potentials into discrete tokens, analogous to spiking in SNNs but with lower training complexity.
Retokenization (Triggering): A neural event is triggered only when the assigned code flips from the previous state ( $z_{im} \neq z_{im-1}$ ). This acts as an asynchronous, many-to-one semantic compressor, discarding redundant events where the code remains constant.

3. Representation Learning (Pre-training)

To stabilize code assignments and prevent "chatter" (frequent flipping at decision boundaries), the authors employ an unsupervised pre-training strategy combining three objectives:

Reconstruction Loss ( $L_R$ ): The discrete codes are aggregated per patch and decoded to reconstruct a time-surface representation of the original events.
Rate Alignment Loss ( $L_{RA}$ ): Encourages similarity in latent representations for events with similar inter-event rates, aligning logits to the event rate dynamics.
Latent Straightening Loss ( $L_{LS}$ ): Minimizes the angular difference in the velocity of subsequent logits, smoothing trajectories in the probability simplex to reduce unnecessary code flips.

Key Contributions

Asynchronous Discrete Encoder: A novel architecture that learns neural events from raw streams, producing outputs that are semantically rich, spatio-temporally sparse, and occupy a compact discrete latent space.
Unique Tokenization Mechanism: A trigger strategy that emits a neural event only upon a code flip. This generalizes level-based sampling to the probability simplex, effectively filtering redundant information.
Stabilizing Pre-training Strategy: A combination of reconstruction, rate alignment, and latent straightness objectives that stabilizes codebook assignment, prevents redundant spiking, and enables high event rate reduction.

Experimental Results

The method was evaluated on object detection (DSEC, Gen1) and classification (N-Caltech101) tasks.

Event Rate Reduction: The framework reduces the event rate by a factor of 2.0 while adding only a single bit of extra information per event (the discrete code index).
Performance:
- Object Detection: On the Gen1 dataset, the proposed method (TokSwinT) achieved 49.9 mAP, surpassing state-of-the-art synchronous (GET, 47.9 mAP) and asynchronous-to-synchronous (EVA-L, 47.7 mAP) approaches. On DSEC, pairing the tokenizer with an asynchronous backbone (TokDAGr) improved the base model's performance by 9.0 mAP.
- Object Classification: On N-Caltech101, the method achieved 86.5% accuracy, performing on par with or better than existing synchronous, asynchronous, and A2S baselines.
Efficiency:
- The encoder requires only 140 KFLOPS per event, which is 17× lower than the state-of-the-art EVA-L encoder.
- The method significantly reduces computational load and energy consumption (µJ/ev) compared to both synchronous and asynchronous baselines.

Significance and Claims

The paper claims that neural events offer a scalable solution to the bandwidth and memory bottlenecks of current event-based vision. By compressing raw streams into discrete tokens, the method retains the critical spatio-temporal dynamics required for high-performance perception while drastically reducing the computational load on downstream networks.

The authors emphasize that this approach enables the use of smaller, more efficient downstream networks and facilitates deployment on memory-constrained edge hardware. While acknowledging a performance gap compared to heavily parameterized synchronous frame-based methods on complex tasks, the work demonstrates that discrete asynchronous encoding can achieve state-of-the-art results with significantly lower resource requirements. The paper concludes that this direction is key to enabling ultra-low-power deployment on edge devices and opens the door to more general forms of event representation learning beyond traditional event cameras.