💻 computer science

Event2Vec: Processing Neuromorphic Events Directly by Representations in Vector Space

The paper introduces Event2Vec, a novel word-to-vector-inspired representation that enables the direct processing of sparse, asynchronous neuromorphic events within high-throughput Transformer architectures, effectively resolving the trade-off between data sparsity and GPU efficiency while achieving state-of-the-art performance with high parameter efficiency and low latency.

Original authors: Wei Fang, Priyadarshini Panda

Published 2026-02-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wei Fang, Priyadarshini Panda

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a conversation, but instead of hearing full sentences spoken at a steady pace, you are listening to a chaotic stream of whispers. Some people whisper quickly, others slowly, and they only speak when something interesting happens. This is how neuromorphic event cameras work. Unlike traditional cameras that take a steady stream of photos (like a flipbook), event cameras only "speak" when a pixel detects a change in brightness. They are incredibly fast, use very little power, and can see in extreme lighting, but their data is messy, sparse, and asynchronous.

The problem is that our current AI "brains" (deep learning models) are like librarians who only know how to read neatly organized books. They struggle with this chaotic stream of whispers.

The Old Ways: Two Flawed Solutions

Before this paper, researchers tried to fix this in two ways, both of which had major downsides:

The "Mosaic" Approach: They tried to glue the whispers together to create a full "sentence" (a dense image frame) so the AI could read it.
- The Flaw: It's like trying to understand a fast-paced conversation by only looking at a blurry photo of the room every second. You lose the speed and the specific timing of the whispers.
The "Specialist" Approach: They built custom AI models designed specifically for messy data (like Spiking Neural Networks).
- The Flaw: These models are like trying to run a marathon on a bicycle. They work, but they can't use the super-fast, powerful engines (GPUs) that modern computers have, making them slow and inefficient.

The New Idea: Event2Vec (The "Translator")

The authors of this paper came up with a brilliant analogy: What if we treated these event whispers like words in a sentence?

Think of a sentence: "How are you?"

Each word has a unique ID (like a dictionary number).
Each word has a position (1st, 2nd, 3rd).
The meaning of a word depends on the words around it.

The authors realized that events work the same way:

An event has a unique ID (its location on the sensor and whether the light got brighter or darker).
An event has a position (its exact timestamp).
The meaning of an event depends on the other events happening nearby in time and space.

They created a system called Event2Vec (Event-to-Vector). Instead of forcing the data into a photo or building a slow, custom engine, they translate every single event into a mathematical "vector" (a list of numbers), just like how modern AI translates words into numbers to understand language.

How It Works (The Magic Ingredients)

To make this translation work perfectly, they added two special ingredients:

The "Neighborhood" Map (Spatial Embedding): In a dictionary, the word "cat" and "bat" might have random numbers next to them, so the AI has to learn they are similar. But in an image, a pixel next to another pixel is always related. The authors built a special map that tells the AI, "Hey, these two pixels are neighbors, so their numbers should be similar." This helps the AI understand shapes and edges much faster.
The "Rhythm" Tracker (Temporal Embedding): Instead of just looking at when an event happened, the AI looks at the gap between events. It's like listening to the rhythm of a drumbeat rather than just the time on a clock. This helps the AI understand speed and motion.

The Results: Fast, Small, and Accurate

The paper tested this new method on three different tasks: recognizing hand gestures, reading sign language, and lip-reading. Here is what they found:

Speed: Because it speaks the same language as modern AI (Transformers), it can use the full power of computer chips. It is 4 to 60 times faster at processing data than previous top methods.
Efficiency: The model is incredibly small. In some cases, it uses 800 times fewer parameters (memory size) than other top models while still getting the job done.
Robustness: It works even if the camera is low-resolution or if there are very few events (whispers). It can still figure out what is happening, whereas other methods fail when the data gets too sparse.
Real-time Ready: It is fast enough to run on small, battery-powered devices (like a robot or a drone) without needing a massive server farm.

The Big Picture

The paper claims that Event2Vec solves a long-standing conflict: it allows us to keep the raw, fast, and sparse nature of event camera data while using the super-fast, powerful AI architectures we already have. It's like finally giving the chaotic whispers a dictionary and a grammar book, allowing them to be understood instantly by the smartest computers in the world.

The code for this system is available for others to use, marking a new way to process visual information that is both efficient and powerful.

Technical Summary: Event2Vec

Problem Statement

Neuromorphic event cameras offer superior temporal resolution, power efficiency, and dynamic range compared to traditional frame-based cameras. However, their data output is inherently asynchronous, sparse, and irregular, typically encoded in Address-Event Representation (AER) as tuples $(x, y, t, p)$ . This format poses a fundamental incompatibility with modern deep learning architectures, which are designed for dense, synchronous, and regularly structured tensors.

Existing solutions face a trade-off:

Dense Representations: Converting events into dense frames (e.g., event frames, voxel grids) allows the use of standard CNNs and Transformers but sacrifices the asynchronous, sparse, and high-temporal-resolution characteristics of the raw data.
Irregular Models: Utilizing Spiking Neural Networks (SNNs), Sparse Convolutions, or Graph Neural Networks (GNNs) preserves sparsity but often fails to leverage the massive parallel processing capabilities of GPUs, leading to lower throughput and higher latency.

The core challenge is to process sparse, irregular event data directly using high-throughput, GPU-accelerated architectures (like Transformers) without converting the data into dense intermediate representations.

Methodology: Event2Vec

Inspired by the analogy between words in Natural Language Processing (NLP) and events in vision, the authors propose Event2Vec, a method that embeds events directly into a vector space.

1. Conceptual Analogy

The paper draws a parallel between NLP tokens and neuromorphic events:

Index: A word's vocabulary index corresponds to an event's spatial address $(x, y, p)$ .
Position: A word's sequential position in a sentence corresponds to an event's timestamp $t$ .
Context: Just as a word's meaning depends on surrounding text, an event's significance is derived from its spatiotemporal context.

2. Vector Representation

The core of Event2Vec is the embedding of an event tuple $(x, y, t, p)$ into a $D$ -dimensional vector $v$ via additive fusion:
$v = v_s + v_t = \text{Embed}_s(x, y, p) + \text{Embed}_t(t)$

Spatial Embedding ( $\text{Embed}_s$ ): Unlike standard NLP look-up tables which treat indices as non-semantic identifiers, the authors propose a parametric spatial embedding. A continuous, differentiable neural network $\phi$ generates the embedding matrix. This ensures that events with spatially adjacent coordinates yield semantically similar vectors, capturing the "neighborhood semantics" critical for visual data.
Temporal Embedding ( $\text{Embed}_t$ ): Instead of using absolute timestamps or sinusoidal encodings (designed for uniform intervals), the temporal module processes the first-order temporal differences ( $\Delta t$ ) between consecutive events. This is implemented via a stack of convolutional layers. This approach provides time-shift invariance, contextual consistency, and acts as a form of temporal preconditioning, allowing the network to learn local event density and velocity.

3. Sampling and Aggregation

To handle variable-length event streams and batch processing requirements:

Random Sampling: Used for simpler tasks, offering high computational efficiency.
K-Means++ Clustering: For complex tasks (e.g., DVS-Lip), events are aggregated into $L$ representative clusters. The authors introduce a GPU-based Batched K-Means++ algorithm to minimize inference latency. The intensity of a cluster (number of raw events aggregated) is used to modulate the embedding vector via a logarithmic weighting factor $(\log \rho + 1)$ .

4. Network Architecture

The Event2Vec representation is fed into a Transformer-based backbone (specifically utilizing the Forgetting Transformer or Gated Linear Attention) followed by a classification head. The architecture includes:

Event2Vec Module: Generates the spatiotemporal embeddings.
Backbone: Stacked Transformer blocks with self-attention mechanisms. The authors extend the Forgetting Transformer to a parameter-shared bi-directional formulation to enhance learning capabilities.
Classification Head: Average pooling followed by a linear layer.

Key Contributions

Direct Sparse Processing: Event2Vec enables the direct processing of sparse, asynchronous event streams using dense, regular Transformer architectures, resolving the conflict between data sparsity and GPU efficiency.
Novel Embeddings: The introduction of parametric spatial embeddings to capture neighborhood semantics and convolution-based temporal embeddings on $\Delta t$ to handle non-uniform timestamps.
Efficient Clustering: A GPU-accelerated Batched K-Means++ algorithm for aggregating events, balancing information retention with low latency.
Paradigm Shift: Demonstrating that event data can be treated as a sequential language, opening the door to leveraging Large Language Model (LLM) architectures for neuromorphic vision.

Experimental Results

The method was evaluated on three benchmarks: DVS Gesture, ASL-DVS, and DVS-Lip.

Accuracy: Event2Vec achieved competitive or state-of-the-art accuracy. Notably, it reached 99.91% on ASL-DVS and 75.88% on DVS-Lip (using clustered events), outperforming several dense-frame and irregular-model baselines.
Parameter Efficiency: The models are significantly smaller than prior SOTA methods. For instance, on the ASL-DVS dataset, the Event2Vec model used 815.93× fewer parameters than a competing GNN & Transformer model.
Throughput and Latency: Event2Vec fully leverages GPU parallelism. Training throughput was up to 11.96× higher, and inference throughput up to 62.67× higher than previous works. Total inference latency was reduced to as low as 11.12% of prior SOTA methods.
Robustness: The model demonstrated strong robustness to low event counts and reduced spatial resolutions (down to 1x1), maintaining accuracy significantly above random guessing.
Ablation Studies: Confirmed that the combination of parametric spatial embedding and convolutional temporal embedding yields the highest accuracy, validating the necessity of inductive biases for neighborhood semantics.

Significance

The paper claims that Event2Vec introduces a novel paradigm for neuromorphic vision by successfully integrating sparse, irregular event data into high-throughput Transformer architectures for the first time. This breakthrough resolves the long-standing trade-off between maintaining data sparsity and maximizing GPU efficiency.

The significance lies not only in the performance metrics (parameter efficiency, low latency, high throughput) which make it suitable for real-time, resource-constrained edge devices, but also in the conceptual alignment of event streams with the NLP paradigm. By treating events as a sequential language, the work suggests new avenues for applying sophisticated architectures developed for Large Language Models to neuromorphic vision tasks.