💻 computer science

InterFuserDVS: Event-Enhanced Sensor Fusion for Safe RL-Based Decision Making

This paper proposes InterFuserDVS, an enhanced sensor fusion architecture that integrates Dynamic Vision Sensors (DVS) with RGB and LiDAR data via a novel token-based strategy to improve the robustness and safety of RL-based autonomous driving agents, achieving a 100% route completion rate on the CARLA Leaderboard.

Original authors: Mustafa Sakhaia, Kaung Sithua, Min Khant Soe Okea, Maciej Wielgosza

Published 2026-05-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mustafa Sakhaia, Kaung Sithua, Min Khant Soe Okea, Maciej Wielgosza

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to drive a car through a busy city. The robot needs to "see" the world to make safe decisions. Usually, we give robots two main types of eyes:

Standard Cameras (RGB): These look like human eyes or phone cameras. They take pictures at a steady pace (like a flipbook). But if the car moves too fast, the pictures get blurry. If the robot drives out of a dark tunnel into bright sunlight, the camera gets "blinded" by the glare, just like your eyes do.
LiDAR: This is like a bat using sonar. It shoots out laser beams to measure distance. It's great for knowing how far away things are, but it can get confused by heavy rain or fog, and it doesn't see colors or details well.

The authors of this paper, InterFuserDVS, decided to give the robot a third pair of eyes called a Dynamic Vision Sensor (DVS), or an "event camera."

The "Event Camera" Analogy

Think of a standard camera as a photographer taking a photo every second. If something moves fast between photos, it looks like a blur.

Now, think of the Event Camera as a security guard who only speaks up when something changes.

If the room is still, the guard says nothing.
If a bird flies across the room, the guard immediately shouts, "Movement at 2 o'clock!"
If a car zooms by, the guard shouts, "Fast movement!"

This "guard" (the DVS) doesn't care about the brightness of the room (it works in pitch black or blinding sun) and it reacts in microseconds (millionths of a second). It only reports the changes in the scene, like moving cars or pedestrians stepping out.

How They Put It All Together

The researchers took a smart driving system called InterFuser (which already used standard cameras and LiDAR) and added this new "event guard" to the team.

They built a Transformer (a type of AI brain) that acts like a conductor in an orchestra.

The Standard Cameras play the melody (colors, traffic lights, signs).
The LiDAR plays the bass (distance, shape of the road).
The Event Camera plays the percussion (fast movement, sudden changes).

The conductor (the Transformer) listens to all three instruments at once. If the standard camera is blinded by the sun, the conductor leans heavily on the Event Camera to see the moving cars. If the Event Camera is too noisy, the conductor listens to the LiDAR for distance. They work together to make a single, safe decision on where to steer and how fast to go.

The Safety Net

Even with a super-smart AI, the authors added a Safety Controller. Think of this as a strict human supervisor sitting next to the robot driver.

If the robot tries to run a red light, the supervisor hits the brakes.
If the robot plans a path that looks like it will hit a pedestrian, the supervisor overrides the robot and stops the car immediately.
They even have a rule: if the robot gets stuck at a red light for too long (over 50 seconds), the supervisor allows it to carefully cross the intersection to avoid getting stuck forever.

The Results

The team tested this robot in a very tough virtual city (called CARLA) filled with traffic, pedestrians, and tricky intersections.

The Goal: Finish the route without crashing or breaking rules.
The Result: Their robot finished 100% of the routes without getting stuck. It scored very high on safety and reliability.
Why it worked: The "event camera" helped the robot see moving people and cars much faster than standard cameras could, especially in tricky lighting or when things were moving fast.

What's Next?

The paper also suggests a future upgrade. Currently, the robot converts the "event" shouts into a format it can read easily (like turning the guard's shouts into a written report). In the future, they want to build a brain that can listen to the shouts directly without converting them first. This would make the robot even faster and more energy-efficient, like a human reacting instinctively rather than thinking about it first.

In short: By adding a super-fast, motion-sensitive "event camera" to a robot's eyes and giving it a strict safety supervisor, the authors created a self-driving agent that is incredibly good at navigating busy, tricky city streets without getting lost or crashing.

Technical Summary: InterFuserDVS

Problem Statement

Autonomous driving systems face significant challenges in high-dynamic-range (HDR) scenes and high-speed scenarios due to the inherent limitations of traditional sensor suites. Standard RGB cameras suffer from motion blur and limited dynamic range, often failing in conditions like exiting a dark tunnel into bright sunlight. While LiDAR provides accurate depth, it is sparse and its performance degrades in adverse weather (rain, fog). Furthermore, existing end-to-end learning approaches often struggle with "causal confusion" and lack the geometric robustness required for safety-critical maneuvers in dense traffic. There is a critical need for a sensor fusion architecture that can leverage complementary modalities to ensure robust perception and safe decision-making under adverse lighting and dynamic conditions.

Methodology

The paper proposes InterFuserDVS, an extended architecture of the state-of-the-art InterFuser model that integrates Dynamic Vision Sensors (DVS), or event cameras, as a primary modality alongside RGB cameras and LiDAR.

1. Architecture Overview

The system employs a global Transformer-based backbone to fuse tokenized features from three distinct sensor types:

RGB Cameras: Multi-view inputs (Front, Left, Right) cropped to 800×600 resolution.
LiDAR: Point clouds voxelized into a 2-bin Bird's Eye View (BEV) pseudo-image (224×224).
DVS (Event Cameras): Raw asynchronous events $(x, y, t, p)$ are accumulated into fixed-time windows to generate "event frames." These single-channel frames represent normalized event counts or polarity sums, capturing motion dynamics and edges with microsecond resolution.

2. Feature Extraction and Fusion

Backbones: RGB and DVS inputs are processed by ResNet variants (e.g., ResNet-18/50), while LiDAR uses PointNet++ adapted for BEV.
Cross-Modal Transfer: A key design choice is initializing the DVS encoder weights using a pre-trained RGB backbone. This leverages the spatial correlation between intensity edges (RGB) and temporal edges (DVS) to facilitate rapid convergence and stabilize early training.
Token-Based Fusion: Features are flattened and projected into $D$ -dimensional tokens. Learnable 2D positional embeddings are added to preserve spatial context. All tokens are concatenated into a single sequence and processed by a 6-layer Transformer Encoder with 8 attention heads. This allows the model to perform global reasoning, dynamically weighting sensor importance based on environmental context (e.g., using DVS for motion and LiDAR for depth).

3. Output and Planning

The Transformer Decoder predicts:

Waypoints: A sequence of 2D coordinates for trajectory planning.
Traffic Light State: Classification of light status (Red, Green, Yellow).
Object Density Maps: A 7-channel BEV grid predicting object existence, offset, size, heading, and velocity.

4. Hybrid Safety Controller

To ensure safety, the system integrates a rule-based safety controller that overrides the end-to-end model's PID commands when necessary:

Traffic Light Compliance: A state machine handles red/yellow light detection. A "Forced Crossover" mechanism prevents deadlocks if the vehicle is stuck for >50 seconds.
Collision Avoidance: The controller utilizes predicted object density maps to render a local occupancy grid. It checks the planned trajectory against this grid at multiple future time steps ( $t=0.5s$ to $2.0s$ ). If the safety distance falls below a velocity-dependent threshold, the system triggers emergency braking.
Control Execution: Final steering, throttle, and brake commands are generated via lateral and longitudinal PID controllers, with target speeds adjusted dynamically based on path curvature and obstacle proximity.

Key Contributions

Event-Enhanced Architecture: The integration of a dedicated DVS processing backbone into the InterFuser framework, utilizing a transfer-learning strategy from RGB to DVS to overcome data scarcity and training instability.
Token-Based Fusion Strategy: A unified Transformer Encoder that fuses spatial features from RGB/LiDAR with high-frequency temporal features from DVS, enabling dynamic sensor weighting.
Hybrid Safety Policy: A robust controller combining end-to-end trajectory predictions with heuristic validation (traffic light state machines and model-based collision checking using occupancy maps).
State-of-the-Art Performance: Demonstration of exceptional reliability on the CARLA Leaderboard, achieving perfect route completion in challenging urban environments.

Experimental Results

The method was evaluated on the CARLA Leaderboard (Town05 Long Routes), a benchmark involving complex navigation, lane changes, and dense traffic.

Driving Score (DS): Achieved 77.20, a competitive score among state-of-the-art methods.
Route Completion (RC): Achieved 100%, indicating the agent never got stuck, lost, or timed out. This outperforms several contemporary methods (e.g., InterFuser at 88.23%, ReasonNet at 89.89%) in terms of completion reliability.
Safety Metrics:
- Collision Rate: Extremely low pedestrian collision rate (0.008/km) and competitive vehicle collision rate (0.036/km).
- Infractions: Red light infraction rate of 0.01/km (attributed to the "Forced Crossover" logic to ensure route completion).
Qualitative Analysis: Case studies demonstrated that DVS significantly improves performance in:
- Unprotected Left Turns: Detecting rapid lateral motion of oncoming traffic via high event density.
- Overtaking: Validating "clearance" in opposing lanes by detecting the absence of high-frequency events (approaching cars).
- Emergency Braking: Reacting to pedestrians emerging from occlusions (e.g., behind parked trucks) faster than RGB-only baselines by detecting limb motion contrast before static visual features resolve.

Significance and Claims

The paper claims that InterFuserDVS validates event-based vision as a vital modality rather than a niche sensor for the next generation of autonomous vehicles. The integration of DVS provides two critical advantages:

High Dynamic Range: Ensures robustness in scenarios with harsh shadows or sun glare where RGB cameras fail.
Temporal Resolution: Captures fast-moving objects explicitly, allowing the model to rely on event data for motion planning during dynamic maneuvers.

The authors conclude that while current implementations convert events to frames, the future lies in Asynchronous Event Fusion using Spiking Neural Networks (SNN) and Asynchronous Fusion Transformers (AFT) to fully exploit the microsecond-level motion information and energy efficiency of neuromorphic hardware. The current work serves as a foundational step, demonstrating that treating event streams as high-fidelity temporal inputs significantly enhances the safety and reliability of end-to-end autonomous driving policies.