DeepIPCv3: Event-Aware Multi-Modal Sensor Fusion for Sudden Pedestrian Crossing Avoidance

Technical Summary of DeepIPCv3: Event-Aware Multi-Modal Sensor Fusion for Sudden Pedestrian Crossing Avoidance

Problem Formulation
Current end-to-end autonomous driving systems predominantly rely on frame-based sensors (e.g., RGB cameras), which suffer from inherent perception latency and motion blur during highly dynamic encounters, such as sudden pedestrian crossings. These limitations create a critical safety vulnerability where the system's reaction time is bottlenecked by the sensor's fixed frame rate. While event-based vision sensors (Dynamic Vision Sensors or DVS) offer microsecond-level temporal resolution by asynchronously capturing pixel-intensity changes, effectively integrating these sparse, asynchronous event streams with dense 3D LiDAR point clouds remains an open research challenge. The core problem addressed is the need for a navigation framework that can maintain structural scene awareness while achieving ultra-low-latency responses to sudden dynamic obstacles, regardless of ambient illumination conditions.

Methodology
The authors propose DeepIPCv3, a novel multi-modal autonomous navigation framework designed to synergize the dense 3D spatial geometry of LiDAR with the microsecond-level asynchronous event streams of a DVS. The architecture is divided into two primary pipelines:

1. Multi-Modal Perception and Fusion:

   • Inputs: The system ingests 3D LiDAR point clouds, asynchronous DVS event streams (aggregated into a spatio-temporal voxel grid), and proprioceptive state data (GNSS, IMU, encoders). RGB-D images are included for ablation studies but are not the primary input for the core model.
   • Backbone: The model utilizes PolarNet (pre-trained on KITTI) for LiDAR point cloud segmentation and EfficientNet-B0 as the backbone for parallel encoders to extract high-dimensional spatial and temporal features.
   • Transformer-Inspired Cross-Modal Attention: A key innovation is the introduction of a Transformer-based fusion module. Unlike simple concatenation, this module employs cross-attention mechanisms where the dense LiDAR features act as the spatial anchor (Query), and the DVS event features provide instantaneous dynamic updates (Key and Value). This allows the network to dynamically shift its "attention," prioritizing high-speed dynamic updates from the DVS when a sudden obstacle appears, while maintaining structural context from the LiDAR.

2. Planning and Control:

   • Waypoint Prediction: The fused latent representation is passed through a Gated Recurrent Unit (GRU) loop, concatenated with the vehicle's navigational state, and processed by a Multi-Layer Perceptron (MLP) to predict a sequence of local waypoints.
   • Hybrid Control Policy: The system employs a hybrid control strategy combining traditional Proportional-Integral-Derivative (PID) tracking with direct neural predictions. The PID controller ensures mathematically bounded, smooth trajectory tracking, while the MLP predicts non-linear adjustments (e.g., hard braking or sharp steering) for sudden maneuvers. The final control commands are a weighted combination of both, with the weighting parameter learned via the Modified Gradient Normalization (MGN) algorithm.

Key Contributions

• Multi-Modal Architecture: The proposal of a framework that synergizes LiDAR and DVS inputs, enabling stable point-to-point navigation while significantly improving responsiveness to sudden dynamic obstacles compared to frame-based systems.
• Cross-Modal Fusion Module: The introduction of a Transformer-inspired attention mechanism that dynamically correlates dense 3D spatial geometry with asynchronous event streams, allowing the generation of optimal driving policies that decouple dynamic responsiveness from ambient lighting.
• Rigorous Offline Evaluation: The creation and utilization of a custom multi-modal dataset featuring sudden pedestrian crossings under both well-illuminated (noon) and illumination-degraded (evening) conditions. The framework is evaluated offline to avoid the severe physical risks associated with live testing of such scenarios.

Experimental Results
The framework was evaluated offline on a dataset of 37,855 synchronized frames collected using a WHILL Model C2 robot equipped with a Velodyne HDL-32E LiDAR, an IniVation DAVIS346 DVS, and a Stereolabs ZED 2 RGB-D camera.

• Performance Metrics: DeepIPCv3 achieved state-of-the-art predictive performance, yielding the lowest Average Displacement Error (ADE) for waypoints and Mean Absolute Error (MAE) for steering and cruise controls compared to baselines like TransFuser, AIM-MT, and LMDrive.
• Response Time: The system demonstrated the lowest Response Time Delay (RTD), achieving 0.208 seconds in noon conditions and 0.211 seconds in evening conditions. This indicates the model executes evasive maneuvers within the same 250ms frame window as the expert driver, effectively eliminating the lag caused by motion blur in frame-based sensors.
• Ablation Studies: Results confirmed that the LiDAR + DVS configuration outperforms LiDAR + RGB, as the RGB stream introduces conflicting, blurry data during high-speed events. Furthermore, the Transformer-based fusion module proved superior to static fusion methods, as it adaptively suppresses static features to maximize the weight of dynamic event streams during critical moments.
• Qualitative Behavior: In noon conditions, the system executed reactive evasive steering maneuvers. In evening conditions, where visibility was low, the system prioritized safety by executing full-stop braking maneuvers, demonstrating contextually appropriate reactive behaviors.

Significance and Claims
The paper claims that DeepIPCv3 addresses a critical safety vulnerability in autonomous driving by fundamentally reducing perception latency at the sensory root rather than relying on post-perception control overrides. By effectively eliminating exposure failures and motion blur through LiDAR-DVS fusion, the proposed system enables highly reactive, mathematically bounded evasive maneuvers regardless of ambient illumination. The authors assert that the near-zero temporal lag between expert action and model prediction is an emergent property of the sensor fusion architecture, achieved by prioritizing low-latency event streams to replicate rapid evasive maneuvers before frame-based motion blur occurs. The work concludes that while offline validation is robust, future work must focus on bridging the gap to safe physical deployment through high-fidelity simulation and edge-computing optimization.