Technical Summary: Bridging the Generalization Gap in Adverse Weather Segmentation

Problem Definition

The paper addresses the challenge of pixel-level semantic segmentation in outdoor scenes degraded by five specific weather conditions: blur, darkness, snow, haze, and glare. The work is framed within the context of the 8th UG2+ Workshop (CVPR 2026) Track 2, which utilizes a dataset of 513 training scenes and 38 test scenes captured by fixed outdoor cameras across diverse geographic locations.

The central problem identified is a severe generalization gap. The authors observe that models achieving high validation accuracy often collapse when tested on unseen scenes. For instance, a SegFormer-B5 model achieved 65.7% mIoU on the validation set but dropped to 49.9% on the test set—a 16.1-point gap. This suggests that increasing model capacity alone is insufficient for robustness in small-data adverse weather scenarios. The primary bottleneck is not raw accuracy, but the ability to generalize from the training distribution to unseen test distributions.

Methodology

The authors propose that a carefully designed training recipe is more critical than architectural complexity. Their approach builds upon the SegMAN-S backbone (31M parameters), a hierarchical encoder-decoder architecture combining neighborhood attention and VSSM (Selective State Space Models) blocks.

The methodology focuses on four key components:

1. Domain-Adaptive Initialization:
   The model is initialized with weights pre-trained on ADE20K (150-class indoor/outdoor segmentation). The classification head is reshaped for the 10-class target task, while backbone and decoder weights are retained to transfer semantic features to the weather domain.

2. Per-Stage Feature Recalibration:
   Lightweight adapter modules are injected into each of the four encoder stages. These modules use a bottleneck structure (1×1 projections, depthwise convolution) and a channel recalibration mechanism. The module computes a scaling factor based on global average pooling and a sigmoid activation, allowing the model to amplify robust feature channels and suppress degraded ones. The modules are zero-initialized to start as identity functions, preserving pre-trained features initially.

3. Scene-Balanced Sampling:
   To prevent large scenes (with hundreds of frames) from dominating the training gradients, the authors implement a scene-level sampler. Each training iteration selects a scene first, then randomly draws a single frame from it, ensuring equal gradient contribution across all 513 scenes regardless of their frame count.

4. Targeted Degradation Augmentation:
   A probabilistic pipeline synthesizes weather artifacts on clean images to match the empirical distribution of the test set (Blur 29%, Darkness 26%, Snow 16%, Haze 16%, Glare 13%). Augmentation is applied with a probability of 0.5 only to clean images to avoid double-degradation artifacts.

Inference Strategy:
At test time, the system leverages temporal consistency by averaging softmax probabilities across all frames within a scene. Additionally, the authors employ Model Soup (uniform weight averaging) across checkpoints from the same recipe but different random seeds to improve out-of-distribution robustness.

Key Results

The proposed system achieves 59.9% mIoU on the official test set with only 31M parameters.

• Performance vs. Larger Models: The approach outperforms the much larger SegFormer-B5 (82M parameters) by 10.0 mIoU points on the test set (59.9% vs. 49.9%).
• Generalization Gap: The validation-to-test gap for the proposed method is 6.5 points, significantly smaller than the 15.8-point gap observed in SegFormer-B5.
• Ablation Insights:
  • Initialization: ADE20K pre-training provided the largest single gain (+2.1 mIoU over mixed-only fine-tuning).
  • Recalibration Scope: Extending recalibration to all four stages improved performance by 0.7 mIoU compared to early-stage-only adaptation.
  • Augmentation: Moderate augmentation (prob=0.5) was optimal; aggressive augmentation (prob=1.0) degraded performance by 1.9 mIoU, indicating a distribution mismatch.

Negative Findings

The paper explicitly reports several strategies that failed to improve performance, offering practical guidance for future work:

• Model Scaling: Larger models (SegFormer-B5, SegMAN-L) suffered from overfitting to the validation distribution, resulting in poor test performance.
• Architectural Modifications: Changes to the decoder, including boundary auxiliary losses and scale-gated refinement, degraded performance.
• Frequency-Domain Inputs: Adding a learnable FFT branch offered only marginal benefits (0.5 mIoU) and showed redundancy with degradation augmentation.
• Consistency Losses: Feature consistency losses (cosine, KL divergence) between clean and degraded pairs underperformed standard cross-entropy loss.
• Restoration Preprocessing: Adding a restoration stem before segmentation was counterproductive, suggesting the model should learn degradation-robust features internally rather than relying on preprocessing.

Significance and Claims

The paper claims that for small-data adverse weather segmentation, training recipe design is more important than model scaling or architectural innovation. The authors argue that the validation-to-test generalization gap is the dominant bottleneck in this domain.

Their primary contribution is a systematic analysis demonstrating that a combination of domain-adaptive initialization, lightweight feature recalibration, scene-balanced sampling, and targeted augmentation produces models that generalize more reliably than larger, more complex alternatives. The extensive analysis of negative results is presented as a practical guide for the community, emphasizing that under limited data conditions, optimizing the training process yields superior robustness compared to increasing model capacity.