💻 computer science

EventDrive: Event Cameras for Vision-Language Driving Intelligence

EventDrive introduces a comprehensive benchmark and a novel vision-language model that leverages the high temporal precision and dynamic range of event cameras to significantly enhance autonomous driving capabilities across perception, understanding, prediction, and planning tasks.

Original authors: Dongyue Lu, Rong Li, Ao Liang, Lingdong Kong, Wei Yin, Lai Xing Ng, Benoit R. Cottereau, Camille Simon Chane, Wei Tsang Ooi

Published 2026-06-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dongyue Lu, Rong Li, Ao Liang, Lingdong Kong, Wei Yin, Lai Xing Ng, Benoit R. Cottereau, Camille Simon Chane, Wei Tsang Ooi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are driving a car, but instead of just taking photos of the road like a normal camera, your car also has a special "motion sensor" that only sees things when they move or change brightness. This is called an Event Camera.

The paper introduces EventDrive, which is like a massive, super-smart driving school curriculum designed to teach computers how to use both regular photos (RGB) and these motion sensors (Events) together to drive safely.

Here is the breakdown of what they did, using simple analogies:

1. The Problem: The "Blurry Photo" vs. The "Motion Sensor"

Regular Cameras (RGB): Think of these like taking a standard photo. If you take a picture of a fast-moving car in the rain or at night, the photo comes out blurry or dark. The camera misses the details because it waits for a fixed amount of time to capture the image.
Event Cameras: These are like a high-speed microphone that only "hears" when something moves. If a car zooms past, the event camera sees the motion instantly, even in total darkness or heavy rain. It doesn't take "photos"; it records tiny, micro-second changes in light.
The Gap: Scientists knew event cameras were great for seeing motion, but they didn't have a way to teach computers to use them for thinking and deciding (like planning a turn or guessing what a pedestrian will do next). Most existing AI could only use them for simple tasks like "is there a car here?"

2. The Solution: The "EventDrive" Textbook

The researchers built a giant dataset called EventDrive. Imagine this as a massive library of driving lessons.

The Content: It contains over 470,000 examples where the computer sees:
1. A regular photo.
2. The motion data from the event camera.
3. A human-written description or question about what's happening.
The Four Levels of Learning: They organized the lessons into four stages, just like a human driver learns:
1. Perception (Seeing the Scene): "Is it raining? Is it night? Is the road wet?" (Event cameras help here because they see edges clearly even when the photo is dark).
2. Understanding (Knowing the Objects): "That's a white van, and it's moving fast." (Event cameras help figure out how fast something is moving, which a blurry photo can't do).
3. Prediction (Guessing the Future): "That car is about to turn left." (Because event cameras see motion so precisely, they can predict movement better than regular cameras).
4. Planning (Making a Decision): "I need to slow down and steer right." (The AI combines the motion data with the scene to decide what to do next).

3. The Teacher: "EventDrive-VLM"

To teach the computer, they built a new AI model called EventDrive-VLM. Think of this model as a student with two sets of eyes and a special brain:

The "Multi-Speed" Brain: The model has a special module that looks at the motion data at different speeds. If the car is moving slowly, it looks at the data in a "slow-motion" way to be stable. If the car is zooming, it switches to "high-speed" mode to catch every tiny movement.
The "Translator": The model has a translator that turns the raw motion data (which is just a stream of dots) into language the AI understands, so it can answer questions like "Is the light red?" or "Where is the pedestrian?"

4. The Results: Why Mixing Them is Better

The paper tested this new system against others:

Regular Cameras alone: Good at recognizing colors and signs in the daytime, but they get confused and make mistakes when it's dark, rainy, or things are moving fast (blur).
Event Cameras alone: Great at seeing motion and speed, but they are "blind" to colors and details (like "is that a red truck or a blue truck?").
EventDrive-VLM (The Mix): By combining both, the AI gets the best of both worlds. It can see the red truck (from the photo) and know exactly how fast it's moving (from the event sensor).
- The Analogy: It's like having a driver who can see clearly in the dark (Event) but also knows what the traffic signs say (Photo). The result is a driver that is much safer and more reliable in bad weather or high-speed situations.

Summary

The paper claims that by creating a huge dataset and a new AI model that mixes photos (for details) with event sensors (for motion and speed), they have created a system that understands driving much better than systems using just one type of sensor. This makes self-driving cars more robust, especially in tricky situations like night driving, heavy rain, or when things are moving very fast.

Technical Summary: EventDrive

Problem Statement

Event cameras offer microsecond temporal precision, high dynamic range, and robustness to motion blur, making them theoretically ideal for autonomous driving in challenging conditions (e.g., low light, rapid motion). However, existing event-based research remains fragmented, focusing primarily on low-level supervised tasks like detection, segmentation, or optical flow estimation. There is a significant gap in utilizing event sensing for high-level reasoning, decision-making, and the full driving loop. While the RGB community has advanced unified vision-language models (VLMs) that couple perception, reasoning, and control, early attempts to incorporate events into VLMs have been limited to generic scenes and simple captioning, failing to address the complex reasoning and decision demands of real-world driving.

Methodology

1. EventDrive: A Unified Benchmark

The authors introduce EventDrive, the first full-stack benchmark integrating event streams, synchronized RGB frames, and language supervision across the entire autonomous driving pipeline. The benchmark decomposes driving into four sequential reasoning stages, each formulated as language-grounded tasks:

Perception: Evaluates scene-level context (scene type, visibility, traffic flow, weather, traffic lights, road conditions).
Understanding: Assesses object-centric semantics (presence, appearance, motion state, ego/environment relations, grounding).
Prediction: Infers short-horizon motion intent (speed and direction) of surrounding agents.
Planning: Estimates ego-vehicle intent and future waypoints based on past trajectories.

Dataset Construction:
EventDrive aggregates 471,543 event–frame–language samples from three source datasets: DSEC, M3ED, and PKU-DAVIS-SOD. The authors employ a semi-automatic pipeline using Qwen3-VL to generate structured captions and convert them into diverse multiple-choice QA pairs, grounding tasks, and trajectory forecasts. A "Hard" split is specifically curated to include sequences with low light and motion blur to rigorously test event sensing advantages.

2. EventDrive-VLM: A Unified Model

To leverage this benchmark, the authors propose EventDrive-VLM, an event-driven VLM designed to interpret and reason over asynchronous event data. The architecture addresses two core challenges:

Dynamic Horizon Event Encoder: Event streams vary widely in temporal density. The model employs a Multi-Horizon Voxelization strategy, creating event tensors at multiple temporal resolutions ( $B = \{20, 50, 100\}$ bins). A Mixture-of-Experts (MoE) gating mechanism dynamically selects the most informative temporal horizon for a given scene, allowing the model to capture high-frequency dynamics during fast motion while ensuring stable aggregation in low-motion regimes.
Event Q-Former Alignment: To bridge the gap between sparse event features and the LLM's semantic space, the model uses an Event Q-Former. Instead of concatenating tokens, it employs cross-attention between learnable event queries and the aggregated event features. This extracts motion-aware, language-aligned tokens that integrate cleanly with frame and text embeddings.
Two-Stage Training Curriculum:
1. Event-Language Pre-adaptation: The LLM and RGB visual encoder are frozen. The event encoder, Q-Former, and projection layers are trained on caption data to align event representations with the LLM space.
2. Instruction Tuning: The LLM transformer blocks are unfrozen, and the entire event pathway is fine-tuned on all caption and QA data to enable coherent multimodal reasoning across all driving tasks.

Key Contributions

EventDrive Benchmark: The first unified dataset and benchmark for autonomous driving that covers perception, understanding, prediction, and planning within a consistent multimodal (event-frame-language) framework.
EventDrive-VLM: A general training framework that equips large VLMs with the ability to interpret, align, and reason over asynchronous event representations using dynamic horizon encoding and query-based alignment.
Comprehensive Evaluation Protocol: A rigorous evaluation demonstrating how temporal cues improve multimodal reasoning, establishing a foundation for event-driven intelligence.

Experimental Results

Experiments on EventDrive reveal distinct modality strengths and the necessity of fusion:

Frame-Only Models: Perform well in perception under normal conditions but degrade sharply in low-light and motion-blur scenarios. They show limited spatial reasoning and weak motion inference (e.g., low accuracy in speed/direction prediction).
Event-Only Models: Excel in speed and direction prediction due to high-frequency temporal cues but lack the semantic richness required for appearance-heavy understanding (e.g., scene type, object appearance).
EventDrive-VLM (Fusion): The event-frame fusion model achieves state-of-the-art performance across all task families. It stabilizes perception under adverse conditions, significantly enhances grounding and relational reasoning, and delivers superior motion prediction and ego-intent estimation compared to single-modality baselines.
Hard Split Performance: In challenging subsets (low light, heavy blur), the fused model maintains robust performance, whereas frame-only models suffer significant drops in grounding and spatial reasoning, and event-only models struggle with semantic tasks.

Significance

The paper claims that EventDrive and EventDrive-VLM demonstrate that event sensing is not merely an auxiliary sensor but a critical modality for reliable driving intelligence. By integrating asynchronous event cues with RGB frames and language, the framework enables robust perception and decision-making in scenarios where conventional sensors fail. The work bridges the gap between low-level event sensing and high-level autonomous driving reasoning, providing a scalable path toward safer, more interpretable, and robust autonomous systems. The authors emphasize that the integration of temporal and spatial modalities yields more reliable reasoning than either modality alone, particularly in dynamic and adverse environments.