DriveXQA: Cross-modal Visual Question Answering for Adverse Driving Scene Understanding

1. Problem Statement

Autonomous vehicles (AVs) require robust perception to navigate complex and adverse environments. While Multimodal Large Language Models (MLLMs) have succeeded in general vision-language tasks, they face significant limitations in safety-critical driving scenarios:

• Lack of Adverse Condition Handling: Existing datasets and models primarily focus on normal conditions, lacking systematic coverage of adverse weather (fog, rain, night) and sensor failures (motion blur, overexposure, LiDAR jitter).
• Inadequate Sensor Fusion: Current MLLM frameworks typically rely on single-modality inputs (mostly RGB) or simplistic fusion mechanisms. They fail to effectively leverage complementary modalities (e.g., using LiDAR to compensate for a foggy camera) to maintain robustness when individual sensors degrade.
• Missing Hierarchical Reasoning: Existing Visual Question Answering (VQA) systems often lack a structured approach to reasoning across different semantic levels (global scene, object relationships, and ego-vehicle status).

2. Methodology

The authors propose a two-pronged solution: a comprehensive dataset (DRIVEXQA) and a novel architecture (MVX-LLM).

A. DRIVEXQA Dataset

• Scale & Composition: Contains 7,885 driving frames and 102,505 QA pairs.
• Adverse Conditions: Systematically covers five weather conditions (Rainy, Night, Sunny, Cloudy, Foggy) and five sensor failure types:
  • Motion Blur (MB)
  • Overexposure (OE)
  • Underexposure (UE)
  • LiDAR Jitter (LJ)
  • Event Low-resolution (EL)
• Hierarchical Taxonomy: Questions are categorized into three levels to ensure comprehensive evaluation:
  1. Global Scene Level: Assessing weather, visibility, and traffic density.
  2. Allocentric Level: Analyzing spatial relationships, distances, and object categorization relative to the scene.
  3. Ego-Vehicle Centric Level: Focusing on lane positioning, sensor health, and immediate decision-making.
• Modalities: Includes four visual modalities (RGB, Depth, Event cameras from 4 views) and LiDAR point clouds.

B. MVX-LLM Architecture

To address the challenge of fusing heterogeneous modalities under sensor degradation, the authors propose MVX-LLM, a token-efficient architecture featuring:

• Specialized Encoders:
  • A shared CLIP-based Vision Transformer for RGB, Depth, and Event camera data.
  • A PointNet++ encoder for LiDAR point clouds (processed separately to handle 3D geometry).
• Dual Cross-Attention (DCA) Projector:
  • This is the core innovation designed to replace standard concatenation or simple projection.
  • It operates via two pathways to enable adaptive feature weighting based on sensor reliability:
    1. Spatial Cross-Attention: Establishes spatial correspondences between modalities (e.g., aligning RGB pixels with LiDAR points).
    2. Channel Cross-Attention: Facilitates semantic information exchange across feature channels.
  • This mechanism allows the model to dynamically weigh reliable signals (e.g., LiDAR in fog) against degraded ones (e.g., RGB in fog).
• Token Aggregation: The fused features are compressed into compact token embeddings using a Query Attention (QAttn) mechanism, which learns to focus on task-relevant spatial regions, outperforming standard pooling methods.

3. Key Contributions

1. DRIVEXQA Dataset: The first large-scale, hierarchical VQA dataset specifically designed for adverse driving conditions, covering diverse weather, sensor failures, and multi-modal inputs (RGB, Depth, Event, LiDAR).
2. MVX-LLM Framework: A novel architecture that successfully integrates multiple complementary sensor modalities using a Dual Cross-Attention mechanism, addressing the gap where existing MLLMs cannot handle multi-sensor inputs efficiently under degradation.
3. Comprehensive Evaluation: Extensive experiments demonstrating that cross-modal fusion significantly outperforms single-modality or naive fusion approaches, particularly in "corner cases" like fog and sensor failure.

4. Experimental Results

• Performance Metrics: Evaluated using BLEU, ROUGE, METEOR, CIDEr, and a GPT-4o-mini based semantic correctness score (GPTScore).
• Overall Performance: MVX-LLM achieved the highest average GPTScore of 55.5, outperforming the best baseline (GAP + Honeybee projector) by 7.3% and the best token fusion method by 3%.
• Adverse Condition Resilience:
  • Foggy Conditions: MVX-LLM scored 53.5, compared to 25.1 for the baseline (prepend strategy).
  • Sensor Failures: The model showed consistent gains across all sensor failure types (e.g., Motion Blur, Overexposure), proving the effectiveness of the DCA mechanism in compensating for degraded sensors.
• Ablation Studies:
  • Dual Attention: The combination of Spatial and Channel attention was crucial; using them together yielded a score of 50.5, while the full triple mechanism (including QAttn) reached 55.5.
  • Modality Combinations: The study confirmed that RGB is foundational (RGB+Depth+Event = 46.0 vs. Depth+Event without RGB = 18.9). However, the full four-modality fusion (RGB+Depth+Event+LiDAR) provided the optimal performance, validating the synergy of geometric and visual data.

5. Significance

• Safety-Critical Advancement: The work addresses a critical gap in autonomous driving: the ability to reason and explain decisions when primary sensors (cameras) fail due to environmental factors.
• Cross-Modal Compensation: It demonstrates that MLLMs can be engineered to act as "cognitive backups," using reliable sensors (like LiDAR) to compensate for unreliable ones (like cameras in fog), thereby enhancing system safety.
• Benchmarking Standard: DRIVEXQA establishes a new standard for evaluating AV perception systems under realistic, stressful deployment conditions, moving beyond the "clean data" bias of previous benchmarks.
• Future Impact: The proposed framework and dataset pave the way for more robust, explainable, and safe autonomous driving systems capable of operating in the real world's unpredictable conditions.