💻 computer science

Toward Deep Representation Learning for Event-Enhanced Visual Autonomous Perception: the eAP Dataset

This paper introduces the eAP dataset, the largest event-camera dataset for autonomous driving, to bridge the gap in event-enhanced deep representation learning by demonstrating its effectiveness in improving 3D vehicle detection and object time-to-contact estimation under challenging illumination conditions.

Original authors: Jinghang Li, Shichao Li, Qing Lian, Peiliang Li, Xiaozhi Chen, Yi Zhou

Published 2026-03-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jinghang Li, Shichao Li, Qing Lian, Peiliang Li, Xiaozhi Chen, Yi Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are driving a car, but your eyes are struggling. It's either blindingly bright (like driving out of a tunnel into the sun) or pitch black (like a moonless night). Your regular car camera (the "Standard" camera) gets confused: it either washes out the image in a white glare or turns everything into a muddy black blob. This is dangerous for self-driving cars.

Now, imagine your car has a second pair of eyes: Event Cameras. These aren't normal cameras that take pictures like a human does. Instead, they are like hyper-sensitive motion sensors that only "see" things when they move or change brightness. They don't care about the sun or the dark; they only care about change.

This paper introduces a new toolkit to teach self-driving cars how to use these "motion eyes" effectively. Here is the breakdown in simple terms:

1. The Missing Puzzle Piece: The "eAP" Dataset

For a long time, scientists had the "motion eyes" (event cameras) but no good way to teach them. They didn't have enough practice data. It's like having a brilliant student but no textbooks.

The authors created the eAP Dataset. Think of this as a massive, high-quality training gym for self-driving cars.

What's in it? 4.8 hours of driving footage on highways and in cities, covering sunny days, rainy nights, and dark tunnels.
The Secret Sauce: It includes both regular video and the "motion sensor" data, perfectly synchronized. It also has labels telling the computer exactly where cars are and how fast they are coming toward the self-driving car.
Why it matters: Before this, researchers were trying to learn to drive in the dark with a flashlight that barely worked. Now, they have a full map and a flashlight that never blinks.

2. Task One: Seeing Cars in the Dark (3D Detection)

The Problem: When a self-driving car drives out of a tunnel, the sun blinds the regular camera. It can't see the car in front of it.
The Solution: The researchers built a system that fuses the "blinded" regular camera with the "motion-sensing" event camera.

The Analogy: Imagine trying to find a friend in a crowded, foggy room. Your eyes (the regular camera) can't see through the fog. But your ears (the event camera) hear them moving. By combining what you see with what you hear, you can pinpoint exactly where your friend is, even if you can't see their face clearly.
The Result: The new system can spot cars in blinding light or total darkness where normal cameras fail completely.

3. Task Two: The "Time-to-Crash" Calculator (TTC Estimation)

The Problem: Self-driving cars need to know: "If I don't brake, how many seconds until I hit that car?" This is called Time-to-Collision (TTC). Old methods tried to do this with complex math formulas (like solving a geometry puzzle on the fly), which often failed in messy real-world traffic.
The Solution: The authors created a new AI brain called Garl-TTC.

The Analogy: Instead of doing complex math, imagine watching a balloon get bigger as it floats toward you. You don't need to calculate the wind speed or the balloon's volume. You just notice: "Wow, that balloon is getting huge really fast. I need to move!"
How it works: The AI learns to watch how the size of a car changes in the image. If the car gets bigger quickly, it's coming fast. If it stays the same size, it's far away. The AI learned this "size-change" trick by studying the massive eAP dataset.
The Superpower: This new system is incredibly fast (200 times per second!) and accurate, even in bad weather. It's like having a co-pilot who never blinks and instantly knows when to hit the brakes.

4. The "Teacher" Trick (Knowledge Distillation)

To make the AI even smarter, the researchers used a "Teacher" model (a very advanced AI called SAM).

The Analogy: Imagine a student (the new AI) trying to learn how to draw a car. Instead of guessing, they have a master artist (the Teacher) standing next to them, saying, "No, the wheel should be here, the bumper is curved like this."
The student learns to draw the car's outline perfectly by copying the master. Once the student learns the lesson, they don't need the master anymore. This helps the AI understand exactly where the car ends and the background begins, leading to better crash predictions.

Why This Matters

This paper isn't just about fancy math; it's about safety.

Current cars often get confused by bad lighting (tunnels, night, glare).
This new approach uses "motion eyes" to see clearly in those situations.
The Dataset ensures that other scientists can build on this work, making self-driving cars safer for everyone, everywhere.

In short: The authors built a giant practice field (the dataset) and taught a self-driving car how to use a special pair of motion-sensing glasses to see through the dark and glare, calculate crash risks instantly, and drive much safer than before.

1. Problem Statement

Autonomous driving systems rely heavily on visual perception, but standard frame-based Standard-Dynamic-Range (SDR) cameras suffer from significant limitations in challenging environments, such as motion blur, over/under-exposure, and low-light conditions. While event cameras offer high temporal resolution and High-Dynamic-Range (HDR) capabilities to mitigate these issues, their adoption in deep learning-based autonomous perception has been hindered by a lack of large-scale, diverse datasets.

Existing event datasets are often small, lack object-level 3D bounding box annotations, or do not provide Time-to-Collision (TTC) ground truth. Consequently, there is a gap in:

Event-enhanced 3D Vehicle Detection: No large-scale dataset exists to train data-hungry Bird's Eye View (BEV) detectors using event data.
Event-enhanced TTC Estimation: Current methods rely on hand-crafted geometric models or optimization, which struggle with complex real-world dynamics and cannot leverage deep learning's generalization capabilities due to data scarcity.

2. Methodology

The paper introduces the eAP (event-enhanced Autonomous Perception) dataset and proposes deep learning frameworks for two specific tasks: 3D Vehicle Detection and Object TTC Estimation.

A. The eAP Dataset

Scale & Diversity: The largest event-camera dataset for autonomous driving, comprising 4.8 hours of data across 58 sequences (Highway, Urban, Low-light). It contains over 532,000 annotated 3D bounding boxes and TTC labels.
Sensor Setup: Features a narrow-baseline (3 cm) rig with a high-resolution RGB camera (1920×1200) and a Prophesee EVK4 event camera (1280×720), synchronized with 5 LiDARs for precise ground truth.
Annotation: Uses a pre-labeling pipeline (BEVFusion + Kalman Filter) followed by manual verification. TTC is calculated based on the depth of the nearest object point and relative velocity.
Key Feature: Includes a "HDR subset" with tunnel traversals and night driving to specifically test robustness against extreme illumination.

B. Task 1: Event-Enhanced 3D Vehicle Detection

Approach: The authors adapt a popular BEV-based 3D detection architecture (similar to BEVFusion).
Fusion Strategy: They employ early fusion, concatenating RGB features and event voxel features before the view transformation module.
Architecture:
- Backbone: ResNet-50 extracts features from both modalities.
- View Transform: Projects front-view (FV) features to a dense BEV voxel grid.
- Temporal Modeling: Incorporates historical BEV features (warped via ego-motion) to improve tracking and detection in dynamic scenes.
Goal: To demonstrate that fusing event data improves detection recall and accuracy, particularly in HDR scenarios where RGB-only models fail.

C. Task 2: Event-Enhanced Object TTC Estimation (Garl-TTC)

The authors propose Garl-TTC (Geometry-Aware Representation Learning for object TTC estimation), a novel framework moving away from hand-crafted geometric fitting.

Core Insight: TTC is strongly correlated with the ratio of visual object heights over time. If an object moves closer, its projected height increases.
Network Design:
1. Height-Ratio Network: Instead of regressing TTC directly, the network learns to predict the visual object height ( $\hat{h}$ ) at two time steps. TTC is then derived analytically: $\hat{\tau} = \frac{\Delta t}{1 - \hat{h}_{t1}/\hat{h}_{t2}}$ . This simplifies the non-linear learning task.
2. Knowledge Distillation: To enhance the model's ability to discern object boundaries (crucial for height estimation), a decoder is added during training to predict a high-resolution object foreground mask. This mask is supervised by a Foundation Model (SAM - Segment Anything Model) via knowledge distillation.
3. Inference: The decoder and distillation loss are removed at inference, maintaining high speed.
Fusion: Uses a late-fusion strategy for the height-ratio network, allowing the model to learn modality-specific features before combining them.

3. Key Contributions

eAP Dataset: The first large-scale, multi-modal dataset with event cameras, 3D bounding boxes, and object-level TTC ground truth, covering diverse lighting and motion dynamics.
First Event-Enhanced 3D Detection: Demonstrates the first successful application of event cameras to improve state-of-the-art BEV 3D vehicle detection, showing significant gains in challenging illumination.
Garl-TTC Framework: Proposes the first deep learning framework for object-level TTC estimation that leverages geometry-aware representation (height ratios) and foundation model distillation.
Real-Time Performance: Achieves 200 FPS on an NVIDIA Orin NX edge device, proving the viability of deep event-based perception for real-time autonomous systems.

4. Results

3D Vehicle Detection

Performance: The event-enhanced multi-modal model outperforms both RGB-only and Event-only baselines in Average Precision (AP) and Average Translation Error (ATE).
HDR Robustness: In High-Dynamic-Range (HDR) scenes (e.g., tunnels), the RGB-only model fails to detect vehicles, while the event-enhanced model maintains high recall.
Generalization: The results hold true across different detector architectures (Anchor-based and CenterNet3D).

TTC Estimation

Accuracy: Garl-TTC achieves a Motion-in-Depth (MiD) error of 45.0 on the eAP test set, significantly outperforming state-of-the-art optimization-based methods (e.g., CMax, ETTCM) which struggle with complex motion and noise.
Safety-Critical Scenarios: The model excels in "crucial" TTC ranges (0–3 seconds), which are most critical for collision avoidance.
Cross-Dataset Generalization: When tested on the EvTTC and FCWD benchmarks without fine-tuning, Garl-TTC outperforms existing methods, demonstrating superior generalization compared to models trained on limited, specific scenarios.
Efficiency: The model runs at 200 FPS (4.55 ms inference time) on an embedded NVIDIA Orin NX, making it suitable for real-time deployment.

5. Significance

Bridging the Data Gap: The eAP dataset addresses the critical bottleneck preventing the adoption of event cameras in deep learning for autonomous driving.
Paradigm Shift: Moves TTC estimation from hand-crafted geometric optimization to data-driven deep representation learning, enabling better handling of complex, non-linear real-world dynamics.
Practical Deployment: By achieving state-of-the-art accuracy with real-time performance on edge hardware, this work validates that event-based deep learning is ready for deployment in safety-critical autonomous systems.
Foundation for Future Research: The dataset and code are open-sourced, providing a benchmark for future research in multi-modal fusion, long-term temporal learning, and closed-loop control.