SB-BEVFusion: Enhancing the Robustness against Sensor Malfunction and Corruptions

Technical Summary: SB-BEVFusion

Problem Statement
Multimodal sensor fusion, specifically combining camera and LiDAR data, has established state-of-the-art performance in 3D object detection for autonomous vehicles by leveraging complementary visual and geometric information. Existing frameworks, such as BEVFusion, typically transform independent sensor data into a unified Bird's-Eye View (BEV) representation for fusion. However, these methods generally assume the presence of high-quality data from all sensors. In real-world scenarios, systems face significant performance deterioration when sensor data is missing, corrupted, or noisy due to adverse weather (e.g., fog, rain), sensor malfunctions (e.g., LiDAR beam reduction), or spatial/temporal misalignment. Current unified representation approaches often struggle to maintain robustness under these specific failure conditions.

Methodology
The authors propose SB-BEVFusion (Single-Branch BEVFusion), a framework-agnostic fusion module designed to handle scenarios where one modality is missing or corrupted. The approach modifies the standard BEVFusion pipeline with the following key components:

1. Unified Architecture: The model employs a single downstream network and detection head shared across both modalities. To facilitate this, the channel widths of the camera and LiDAR BEV representations are equalized ($C_{cam} = C_{lid} \triangleq C$).
2. Adaptive Fusion Logic:
   • Dual Modality ($L+C$): When both sensors are available, the BEV features are combined using a fusion operator $\mathcal{F}$. The paper investigates four operators: Unweighted Averaging, Max-Pooling, Cross-Attention, and Progressive Modality Decay (PMD).
   • Single Modality ($L$ or $C$): When one modality is absent or corrupted, the fusion operator acts as an identity function, passing the available BEV representation directly to the encoder without attempting to fuse missing data.
3. Training Strategy: The model is trained on a shuffled mixture of three regimes: LiDAR-only ($L$), Camera-only ($C$), and both ($L+C$). For fusion strategies like Averaging, Max-Pooling, and Cross-Attention, all three regimes are enumerated per sample. For PMD, the training involves an "anchor" modality where the secondary modality is multiplied by a decaying factor $\alpha$ (ranging from 1 to 0) to simulate the transition to missing data.
4. Inference: During inference, the system dynamically switches between the fusion operator (when both modalities are present) and the identity function (when one is missing), requiring no structural changes to the network.

Key Contributions

• Robust Fusion Module: Introduction of a fusion module explicitly designed to operate under $L+C$, $L$, and $C$ scenarios, allowing the model to maintain performance during sensor failure.
• Fusion Strategy Analysis: A comprehensive investigation of various fusion operators (Averaging, Max-Pooling, Cross-Attention, PMD) to determine their robustness against missing and corrupted modalities.
• Empirical Validation: Extensive experiments conducted on the MultiCorrupt dataset (based on nuScenes) demonstrating improved robustness across a wide range of sensor deterioration scenarios, including extreme weather and sensor failures.

Results
The authors evaluated SB-BEVFusion against BEVFusion and UniBEV on the nuScenes validation set using the MultiCorrupt benchmark, which includes corruptions such as beam reduction, spatial/temporal misalignment, fog, and motion blur at three severity levels.

• Corruption Robustness: Under various corruptions, the Unweighted Averaging strategy (SB-BEVFusion-Avg) achieved the highest Mean Resistance Ability (mRA) of 0.7683, outperforming BEVFusion (0.7490) and UniBEV (0.7656). Cross-Attention (SB-BEVFusion-CA) showed superior resilience specifically against spatial and temporal misalignments.
• Missing Modality Performance:
  • LiDAR-Only: SB-BEVFusion significantly outperformed baselines, achieving an mAP of 0.6448 compared to BEVFusion's 0.5639 and UniBEV's 0.582.
  • Camera-Only: While UniBEV demonstrated higher robustness in camera-only settings (mAP 0.35 vs. 0.2002 for SB-BEVFusion), SB-BEVFusion maintained competitive performance in full sensor settings (mAP 0.6737) while offering superior LiDAR-only robustness.
• Qualitative Analysis: Visual comparisons confirmed that under severe corruptions (e.g., fog severity 3, motion blur severity 2), BEVFusion often failed to detect large objects or produced bounding boxes deviating significantly from ground truth, whereas SB-BEVFusion maintained more accurate predictions.

Significance
The paper claims that SB-BEVFusion addresses a critical vulnerability in autonomous perception systems: the reliance on ideal sensor conditions. By demonstrating that a single-branch architecture trained on a mixture of modality availability regimes can effectively handle missing or corrupted data, the work provides a pathway toward more reliable 3D object detection in real-world environments. The authors highlight that simple concatenation (used in standard BEVFusion) is suboptimal for corrupted data and that the choice of fusion operator significantly impacts robustness depending on the specific type of sensor failure. The study concludes that while the proposed method excels in LiDAR-only failure scenarios and general corruption resistance, future work could explore transformer-based fusion methods and the integration of additional modalities like radar.