Technical Summary: IAF-Net for Low-Light Urban Road Segmentation

Problem Statement

Robust semantic road segmentation is a critical component of autonomous driving systems, essential for path planning and obstacle avoidance. While existing methods perform well under standard lighting, they suffer severe performance degradation in low-light and nighttime conditions. The primary limitation identified is not a lack of nighttime studies, but rather the insufficient modeling of illumination-dependent modality reliability.

In low-light scenarios, RGB images suffer from reduced brightness, low contrast, and sensor noise, which corrupt texture and color features. Crucially, many existing multi-modal fusion methods (combining RGB with geometric data like depth or surface normals) fail to explicitly adapt to these changes. Consequently, they often propagate degraded RGB noise into the fused representation, rather than dynamically shifting reliance toward more reliable geometric cues when lighting conditions deteriorate.

Methodology: IAF-Net

The authors propose IAF-Net (Illumination-Adaptive Fusion Network), an end-to-end framework designed to dynamically adjust feature fusion based on scene illumination. The architecture integrates four core modules:

1. Lightweight Low-light Enhancement Module (LLEM):

   • Takes raw low-light RGB images as input.
   • Uses depth-wise separable convolutions to estimate pixel-wise enhancement parameters, iteratively improving brightness and contrast while suppressing noise.
   • Key Output: Produces an enhanced RGB image and a global brightness estimation ($b \in [0, 1]$), which serves as a scene-level condition signal for subsequent modules.

2. Robust Surface Normal Estimation (R-SNE):

   • Processes depth maps to generate surface normal cues, which are inherently less sensitive to illumination changes than RGB appearance.
   • Pipeline: Performs outlier rejection on raw depth (using Z-score filtering), applies bilateral filtering to preserve edges, and computes surface normals.
   • Confidence Weighting: A lightweight network predicts a pixel-wise confidence map to weight the normal features, reducing the contribution of unreliable normal estimates in noisy regions.

3. Illumination-Adaptive Fusion (IAF) Module:

   • The core innovation that links the brightness signal to multi-modal integration.
   • Mechanism: A Multi-Layer Perceptron (MLP) maps the global brightness signal $b$ to dynamic fusion weights ($\alpha_{rgb}$ and $\alpha_{norm}$) that sum to 1.
   • Behavior: In extremely dark scenes (low $b$), the module assigns higher weight to geometric (normal) features and suppresses noisy RGB features. In well-lit scenes, it prioritizes RGB texture information while retaining geometric support.
   • Fusion Process: Features are processed through Hierarchical Attention Modules (HAM), combined using the dynamic weights, and refined via a Heterogeneous Feature Contrast Descriptor (HFCD) and Affinity-Weighted Feature Refiner (AWFR).

4. Nighttime-aware Attention Decoder (NAA):

   • Replaces standard attention gates with a brightness-modulated attention sharpening mechanism.
   • Mechanism: The attention map $\psi$ is sharpened by a factor $s = 1.0 + (1.0 - b) \times 0.5$.
   • Effect: In dark scenes, the sharpening factor increases, forcing the attention mechanism to focus more strictly on relevant features and suppress background noise. In bright scenes, the factor approaches 1, maintaining standard behavior.

Optimization: The framework includes an auxiliary EdgeHead for boundary prediction and an Adaptive Loss Weighting (ALW) scheme. The ALW modulates semantic transition, depth inconsistency, and edge supervision losses based on the global brightness estimate, ensuring the training objective aligns with the illumination-adaptive design.

Key Contributions

1. IAF-Net Framework: An end-to-end network that explicitly integrates illumination awareness into the perception pipeline, adapting to varying lighting conditions without requiring domain transfer or additional sensors.
2. IAF Module: A novel fusion mechanism that uses global brightness to dynamically regulate the reliability of RGB vs. geometric features, effectively suppressing noise propagation from degraded RGB data at night.
3. NAA Decoder: A brightness-modulated attention decoder that enhances feature selection in low-light environments by adaptively sharpening attention maps.
4. Dedicated Datasets:
   • nuScenes-NRS: A large-scale dataset derived from nuScenes with automatically generated pixel-level road segmentation annotations from LiDAR point clouds (3,182 training, 805 validation images).
   • CARLA-MWRS: A synthetic dataset covering multi-weather conditions (sunny day/night, foggy night, heavy rain foggy night) to test robustness (3,600 images total).

Experimental Results

The authors evaluated IAF-Net on the proposed datasets, comparing it against state-of-the-art single-modal and multi-modal methods.

• Performance on nuScenes-NRS: IAF-Net achieved State-of-the-Art (SOTA) performance among compared methods.
  • MaxF Score: 96.11% (outperforming the strong baseline RoadFormer+ by 0.15% and SNE-RoadSegV2 by 0.57%).
  • Efficiency: Despite outperforming RoadFormer+, IAF-Net uses significantly fewer parameters (73.78M vs. 131.3M).
• Robustness on CARLA-MWRS: The model demonstrated stability across adverse weather conditions, achieving a MaxF of 94.79% even in heavy rain foggy nighttime scenarios.
• Ablation Studies: Conducted on a 40% training subset to isolate module contributions.
  • The IAF module provided the largest individual gain (+0.70% MaxF).
  • The NAA module provided the second-largest gain (+0.63% MaxF).
  • The full model improved MaxF by 1.29% over a simplified baseline (R-SNE + backbone).

Significance and Claims

The paper claims that the primary significance of IAF-Net lies in its explicit modeling of illumination-dependent modality reliability. Unlike previous approaches that treat fusion weights as static or implicitly learned, IAF-Net uses a global brightness signal as an explicit control mechanism. This allows the system to automatically rely on geometric features when RGB data is unreliable (night/dark) and leverage texture when it is informative (day/well-lit).

The authors position this work as a step toward more robust autonomous driving perception in adverse conditions, addressing the specific gap where existing fusion methods fail to account for the shifting reliability of RGB cues. The construction of the nuScenes-NRS and CARLA-MWRS datasets is also highlighted as a contribution to facilitate future research in low-light road segmentation.

Future work is identified as focusing on improving efficiency for real-time deployment, extending the framework to more complex scenarios (snow, rain), and integrating tighter with downstream planning and control modules.