Technical Summary: WeatherOcc3D

Problem Statement

3D semantic occupancy prediction is critical for autonomous driving navigation, yet its robustness is fundamentally limited by environmental variability. While multi-modal systems fusing camera and LiDAR data generally enhance performance, they face a "modality trust problem" in adverse conditions. Specifically, camera sensors suffer from severe visibility degradation in heavy precipitation, while LiDAR sensors encounter significant backscatter noise during rain. Existing state-of-the-art fusion strategies, such as attention-based models (e.g., OccFusion, GaussianOcc3D), often fail to adaptively re-weight inputs when a specific sensor becomes unreliable. Furthermore, these methods frequently incur prohibitive memory costs or treat sensor inputs as equally reliable regardless of atmospheric noise. While recent Vision-Language Model (VLM) approaches utilize linguistic embeddings for semantic priors, they rarely address the specific challenge of sensor trust in degraded weather conditions.

Methodology

The authors propose WeatherOcc3D, a VLM-assisted framework designed to guide multi-sensor integration via linguistic environmental cues. The architecture operates as an end-to-end multi-modal system with the following key components:

1. Feature Extraction:

   • Camera: Surround-view images are processed through a 2D encoder (ResNet + FPN) and projected into 3D space using a View Transformer (Lift-Splat-Shoot) to generate camera voxel features ($V_{cam}$).
   • LiDAR: Point clouds are processed through a 3D Encoder to generate LiDAR voxel features ($V_{pts}$).

2. Environmental Perception and Prompting:

   • A weather prediction module analyzes 2D features to predict environmental states: Visibility (Clear/Rainy) and Illumination (Day/Night).
   • These predicted states are converted into text prompts (e.g., "Rainy night, heavy camera glare").
   • A pre-trained CLIP text encoder (specialized via Low-Rank Adaptation/LoRA) encodes these prompts into a refined environment embedding ($f_{env}$). A projection layer maps this embedding to the model's feature dimension.

3. Factorized Gating Mechanism:
   The framework employs a dual-level adaptive fusion strategy driven by $f_{env}$:

   • Channel-Level Gating: The embedding generates gating masks ($G_{cam}, G_{pts}$) applied via element-wise multiplication. This allows the model to selectively suppress noise-contaminated channels within each modality based on the specific environmental context.
   • Global Weighting: The embedding is transformed into a single learnable scalar ($w_{env}$) via an MLP and sigmoid activation. This scalar represents the model's global confidence in the visual modality.
   • Fusion: The final fused representation ($V_{fused}$) is computed as a weighted sum:
     $$V_{fused} = w_{env}(G_{cam}V_{cam}) + (1 - w_{env})(G_{pts}V_{pts})$$
     This mechanism enables the model to prioritize semantic camera features in clear daylight while shifting reliance to geometric LiDAR priors during rainy nights.

4. Optimization:
   The model is trained with a multi-task objective function combining occupancy prediction loss (cross-entropy and Lovász-Softmax) and weather prediction loss (binary cross-entropy for visibility and illumination).

Key Contributions

• VLM-Guided Factorized Gating: Unlike memory-intensive cross-attention methods, the paper introduces a parameter-efficient gating mechanism that decomposes environmental uncertainty into independent visibility and illumination factors to dynamically modulate fusion ratios.
• Modality Trust Resolution: The framework explicitly addresses the modality trust problem by using linguistic cues to suppress noise-contaminated channels and adjust global fusion weights, rather than treating sensors as equally reliable.
• Plug-and-Play Versatility: The method is designed as a modular component that can be integrated into existing architectures without requiring full re-architecture.

Experimental Results

Evaluations were conducted on the nuScenes-OpenOccupancy validation set.

• Performance Gains:
  • Integrated with OccMamba, the framework achieved 26.3 mIoU, a 1.1 mIoU improvement over the vanilla baseline.
  • Integrated with M-CONet, it achieved 21.1 mIoU, a 1.0 mIoU improvement.
  • Improvements were consistent across all 17 semantic classes.
• Adverse Weather Robustness:
  • In Night conditions, the method improved mIoU from 11.8 to 15.7 (+3.9).
  • In Rainy conditions, mIoU improved from 24.1 to 27.3 (+3.2).
• Efficiency:
  • The proposed module added only 2.14 ms of latency, outperforming the ACLF method (3.21 ms) while achieving higher segmentation scores (+0.8 mIoU over ACLF).

Significance and Claims

The paper claims that WeatherOcc3D establishes a more robust paradigm for multi-modal perception in autonomous driving. By leveraging the pre-trained CLIP latent space to interpret environmental descriptors, the framework provides a computationally efficient solution to the modality trust problem. The authors emphasize that their approach effectively mitigates sensor-specific degradation, particularly in the most challenging adverse weather subsets, offering a significant improvement over traditional static fusion baselines without the high memory overhead of attention-based alternatives. The work demonstrates that dynamically adapting fusion weights based on environmental visibility and illumination is essential for reliable 3D semantic occupancy prediction in real-world scenarios.