💻 computer science

Degradation-Aware Modality Trust Estimation for Camera--LiDAR BEV Fusion in 3D Object Detection

This paper proposes DARFusion, a lightweight framework that enhances camera-LiDAR BEV fusion for 3D object detection by estimating modality trust through degradation-aware reliability conditioning, thereby preserving performance on clean data while significantly improving robustness against various camera degradations.

Original authors: Weifeng Xu, Yaqian Cui, Hongtao Wang

Published 2026-10-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Weifeng Xu, Yaqian Cui, Hongtao Wang

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Autonomous vehicles navigate the world by building a mental map of their surroundings, relying on a team of sensors to see what a human driver would. Two of the most important members of this team are cameras and laser scanners. Cameras act like human eyes, capturing rich details about color, texture, and the shape of objects, while laser scanners, known as LiDAR, send out beams of light to measure exact distances and create a precise 3D skeleton of the scene. To make a car drive safely, engineers must combine these two very different views into a single, unified picture. This process, often called sensor fusion, allows the vehicle to understand not just that a car is there, but exactly where it is and how big it is. However, this partnership has a weakness: cameras are vulnerable to the weather. Just as a human's vision can be blurred by rain, fog, or darkness, a camera's view can become distorted by glare, motion blur, or low light. When this happens, the camera sends confusing signals to the car's computer. If the computer blindly trusts these corrupted signals, it might mistake a smear of rain on the lens for a real object, or fail to see a pedestrian in the shadows. The challenge for researchers is to teach the car's brain to know when to trust the camera and when to rely more heavily on the laser scanner, especially when the camera's view is compromised.

A team of researchers from North China Electric Power University has developed a new method to solve this problem, creating a system they call DARFusion. Their approach is built on a simple but powerful idea: instead of just merging the camera and laser data together, the system should first ask, "How much can we trust the camera right now?" In standard systems, the computer often treats the camera's input as equally reliable as the laser's, regardless of whether the image is clear or muddy. This new method changes that dynamic by adding a layer of judgment. It constantly monitors the quality of the visual information coming from the camera. If the camera is seeing a clear day, the system uses the rich visual details to help identify objects. But if the camera is struggling with rain, blur, or darkness, the system learns to dial down the camera's influence and lean more on the laser scanner's stable geometric data. This isn't about turning the camera off; it is about adjusting the volume of its voice so that it doesn't drown out the more reliable information when the view is poor.

To achieve this, the researchers designed a lightweight module that acts as a gatekeeper between the two sensors. Before the camera and laser data are combined, the system analyzes the camera's view to detect signs of degradation. It looks for specific patterns in the image data that suggest the lens is dirty, the light is too dim, or the image is too blurry. It also checks the overall reliability of the laser data. Based on these checks, the system calculates a trust score for the camera at every single point in the scene. This score determines how much of the camera's information should be mixed into the final picture. If the camera is struggling, the system reduces its contribution, preventing the car from making decisions based on visual noise. If the camera is clear, the system allows the full visual detail to enhance the laser data. This process happens in real-time, allowing the vehicle to adapt instantly to changing weather conditions without needing any extra sensors or manual adjustments.

The team tested this new system using two major datasets of real-world driving scenes, one from a large-scale autonomous driving project and another from a well-known research collection. They evaluated the system under perfect conditions and then subjected it to various simulated challenges, such as heavy rain, motion blur, and low-light night driving. In clear weather, the new system performed just as well as the best existing methods, proving that it does not slow down or confuse the car when everything is going smoothly. However, when the conditions worsened, the difference became clear. In tests simulating rainy nights and heavy motion blur, the new system significantly outperformed the standard approach. For instance, in simulated rain, the system improved its ability to detect objects by a noticeable margin compared to the older method. Even in the most difficult simulated night conditions, where the camera's view was severely limited, the system maintained a higher level of accuracy. This suggests that the car's "brain" successfully learned to ignore the confusing visual noise and rely on the laser scanner when the camera was unreliable.

The researchers also verified that this improvement works across different types of vehicles and environments, showing consistent results whether the car was driving in a city or on a highway. They found that the system was particularly good at spotting large objects like trucks and buses, which are critical for safety. The method achieved these results without adding significant computational cost, meaning it could be installed on current self-driving cars without requiring massive new computers. The study suggests that the key to robust autonomous driving is not just having better sensors, but having smarter ways to combine them. By teaching the vehicle to recognize when its visual senses are failing and to adjust its trust accordingly, the system creates a more resilient perception that can handle the unpredictable nature of the real world. This work offers a practical path forward for making self-driving cars safer in the rain, at night, and in other challenging conditions where human vision also struggles.

Technical Summary: Degradation-Aware Modality Trust Estimation for Camera–LiDAR BEV Fusion in 3D Object Detection

1. Problem Statement

Current Camera–LiDAR Bird's-Eye-View (BEV) fusion methods for 3D object detection rely on the assumption that strong feature responses in the camera BEV branch correspond to reliable visual evidence. However, under degraded sensing conditions (e.g., low illumination, blur, noise, or adverse weather), feature activations may stem from sensor artifacts or corrupted textures rather than useful scene content.

Existing fusion strategies, such as direct concatenation, convolutional fusion, or attention mechanisms, are primarily "content-driven." They determine fusion weights based on feature alignment and semantic content but lack explicit mechanisms to assess the reliability of the visual evidence when the camera stream remains active but degraded. Consequently, these methods may inadvertently propagate unreliable, degradation-induced visual patterns into the joint representation, even when the LiDAR modality provides stable geometric information. The core challenge addressed is how to decouple feature representation from modality trust estimation under active camera degradation without requiring explicit degradation labels or image-quality supervision.

2. Methodology: DARFusion

The paper proposes DARFusion (Degradation-Aware and Reliability-Guided Fusion), a lightweight framework that reformulates BEV fusion as a modality trust estimation problem. Instead of fusing raw features directly, DARFusion conditions the fusion decision on reliability cues, task-driven contexts, and degradation descriptors. The pipeline consists of three main components:

2.1 Modality Reliability Calibration (MRC)

Before spatial fusion, MRC globally recalibrates camera ( $F_c$ ) and LiDAR ( $F_l$ ) BEV features.

Process: A Global Average Pooling (GAP) operation generates a compact descriptor $s_m$ for each modality.
Output: Two lightweight branches process $s_m$ $s_{m}$ :
1. A reliability cue $q_m \in [0, 1]$ (via sigmoid) that scales the feature map ( $\tilde{F}_m = q_m \odot F_m$ ).
2. A task-driven auxiliary context $u_m$ (derived from a predicted log-variance) that provides sample-level modality context.
Nature: These are not statistical uncertainty estimates but task-driven cues learned through the detection loss to adjust modality strength.

2.2 Degradation-Aware Conditioning (DAC)

To address active camera degradation where the stream is available but compromised, DAC extracts a compact degradation descriptor ( $d$ ) directly from the calibrated camera BEV features ( $\tilde{F}_c$ ).

Mechanism: The camera feature is projected to a single-channel response map $X$ $X$ . Three response-statistics proxies are computed:
1. Activation-level proxy ( $r_b$ ): Global mean activation (sensitive to illumination).
2. Edge-response proxy ( $r_s$ ): Inverse of mean edge response (sensitive to blur).
3. High-frequency residual proxy ( $r_n$ ): Standard deviation of local residuals (sensitive to noise).
Descriptor: These proxies are mapped to $[0, 1]$ via learnable affine transforms and sigmoids to form $d = [d_b, d_s, d_n]$ . This descriptor exposes the visual sensing state (brightness, blur, noise) to the fusion gate without external supervision.

2.3 Degradation-Conditioned Fusion Gate

The fusion gate predicts spatial weights for the camera and LiDAR branches based on a concatenated input vector $Z$ :
$Z = [\bar{F}_c, \bar{F}_l, u_c, u_l, d]$
Where $\bar{F}$ represents projected calibrated features, $u$ represents auxiliary contexts, and $d$ is the degradation descriptor.

Operation: A lightweight network (two $1\times1$ convolutions with ReLU and sigmoid) outputs gate matrices $G_c$ and $G_l$ (where $G_c + G_l = 1$ ).
Fusion: The final BEV feature is computed as a weighted sum: $F_{fuse} = G_c \odot \bar{F}_c + G_l \odot \bar{F}_l$ .
Logic: The gate learns to reduce the weight of the camera branch when the degradation descriptor indicates poor sensing conditions, relying more on LiDAR geometry.

3. Key Contributions

Problem Formulation: The authors identify and formulate the reliability limitation in content-driven BEV fusion, where feature responses can be entangled with modality trust under degraded sensing.
DARFusion Architecture: A lightweight fusion method that integrates:
- MRC: For global modality feature recalibration.
- DAC: For extracting compact degradation descriptors from BEV features (illumination, blur, noise) without image-quality labels.
- Reliability-Guided Fusion: A gate that uses these cues to estimate modality trust spatially.
Comprehensive Evaluation: A degradation-oriented evaluation on nuScenes and KITTI covering:
- Clean validation sets.
- Real-world rain and night subsets.
- Controlled synthetic degradations (synthetic night, motion blur, Gaussian noise).
- Cross-dataset robustness consistency.

4. Experimental Results

The method was evaluated against a reproduced BEVFusion baseline and other state-of-the-art detectors.

4.1 Clean-Set Performance

DARFusion preserves and slightly improves performance on clean data:

nuScenes: Achieved 68.4 mAP and 71.5 NDS (vs. baseline 67.8 mAP / 71.0 NDS).
KITTI: Improved mean Moderate 3D APR40 from 76.2 to 77.1.
Efficiency: Added only 0.04M parameters (~0.1% overhead) and reduced FPS from 4.57 to 4.46 on an RTX 4090.

4.2 Robustness Under Degradation

DARFusion demonstrated significant gains in adverse conditions:

Real-World Subsets:
- Rain: +1.41 mAP improvement.
- Night: +1.31 mAP improvement.
Synthetic Degradations:
- Synthetic Night: Consistent gains across low, mid, and high severity, with the largest gain (+1.1 mAP) at mid-level degradation.
- Motion Blur: Maintained higher mAP across all blur strengths ( $k=1$ to $21$), with a peak gain of +1.2 mAP.
- Gaussian Noise: Showed improvements at low-to-moderate noise levels (+0.6 mAP), though gains diminished at extreme noise levels ( $\sigma=30$ ) where visual evidence is heavily corrupted.
Cross-Dataset Consistency: The robustness improvements were consistent across both nuScenes and KITTI, with DARFusion outperforming the baseline in all 9 tested corruption-severity settings on KITTI.

4.3 Mechanism Analysis

Ablation: Removing the Degradation-Aware Conditioning (DAC) or the auxiliary contexts (U) resulted in performance drops, confirming the necessity of both reliability cues and degradation descriptors.
Behavioral Analysis: The magnitude-based relative camera contribution ( $R_{cam}$ ) decreased systematically as degradation severity increased. Spearman correlation analysis confirmed a strong negative correlation between degradation severity and camera contribution, and a positive correlation between the learned descriptors and severity.

5. Significance and Claims

The paper claims that explicitly modeling camera reliability is an effective and lightweight strategy to improve BEV fusion robustness.

Key Insight: By reformulating fusion as a trust estimation problem conditioned on degradation descriptors derived from BEV features, the system can distinguish between useful semantic evidence and degradation-induced interference.
Practicality: The method requires no degradation labels or auxiliary image-quality supervision, making it suitable for real-world deployment where such ground truth is unavailable.
Limitations: The authors modestly note that the current study focuses on active camera degradation with unchanged LiDAR. It does not cover coupled sensor failures or LiDAR degradation. Furthermore, while the method handles moderate noise well, extremely dense pixel-level noise remains challenging. The descriptors are task-driven cues rather than calibrated physical measurements.

In conclusion, DARFusion offers a practical pathway to decouple feature representation from modality trust, ensuring that 3D object detection systems remain robust when visual sensing is compromised.