WeatherReasonSeg: A Benchmark for Weather-Aware Reasoning Segmentation in Visual Language Models

1. Problem Statement

While Vision-Language Models (VLMs) have demonstrated impressive capabilities in reasoning-based segmentation (generating segmentation masks based on complex natural language queries rather than simple class labels), current benchmarks rely almost exclusively on high-quality images captured under idealized conditions.

The paper identifies a critical gap: How do VLMs perform when visual cues are severely degraded by adverse weather conditions (rain, snow, fog, low-light)? In real-world safety-critical applications (e.g., autonomous driving), weather-induced degradation (reduced visibility, noise, structural distortion) disrupts the visual-language alignment, potentially causing reasoning pipelines to fail. Existing benchmarks lack systematic evaluation of reasoning segmentation under these specific environmental stressors.

2. Methodology

The authors introduce WeatherReasonSeg, a comprehensive benchmark comprising two complementary components designed to evaluate VLM robustness:

A. Controllable Synthetic Adverse-Weather Dataset

• Base: Built upon the existing ReasonSeg dataset.
• Synthesis Mechanism: Instead of simple noise injection, the authors use physics-based weather synthesis to simulate rain, snow, and fog.
  • Rain: Modeled using region-dependent rain streaks and accumulation effects (simulating heavy rain as thin fog).
  • Fog: Modeled using the atmospheric scattering model, controlled by depth maps and scattering coefficients.
  • Snow: Modeled as a combination of background imaging, particle occlusion layers, and atmospheric scattering.
• Control: Severity is continuously controllable (Light, Moderate, Heavy) by adjusting physical parameters (e.g., rain density, visibility distance, particle scale). This allows for fine-grained analysis of how increasing degradation impacts reasoning stability.

B. Real-World Adverse-Weather Dataset

• Base: Built upon the ACDC (Adverse Conditions Dataset with Correspondences), which contains real-world images with pixel-level annotations.
• Query Generation: A novel mask-guided Large Language Model (LLM) prompting strategy is employed.
  • The LLM is provided with the degraded image, the ground-truth mask, and the object category.
  • It generates five distinct reasoning queries for the same target object, covering five specific dimensions:
    1. Function: Intrinsic purpose of the object.
    2. Application: Contextual usage scenarios.
    3. Structure: Observable attributes and compositional details.
    4. Relationship: Spatial or interactive relationships with other entities.
    5. Requirement: Mapping practical needs to the object.
• Filtering: A rigorous human-model collaborative verification process ensures semantic consistency, single-target alignment, and answerability based solely on the image.

C. Evaluation Framework

The benchmark evaluates diverse VLMs (including Grounded-SAM, LISA, Seg-R1, and Seg-Zero) using Intersection-over-Union (IoU) metrics (gIoU and cIoU) across the synthetic and real-world datasets.

3. Key Contributions

1. First Weather-Aware Reasoning Benchmark: WeatherReasonSeg is the pioneering benchmark specifically designed to bridge the gap between idealized reasoning segmentation evaluation and real-world deployment under adverse weather.
2. Graded Synthetic Dataset: A controllable dataset that quantitatively characterizes the impact of weather intensity (Light to Heavy) on reasoning processes, enabling systematic robustness analysis.
3. Real-World Multi-Dimensional Dataset: A curated real-world dataset with semantically consistent, executable queries generated via mask-guided LLM prompting, covering five complex reasoning dimensions.
4. Comprehensive Analysis: The paper provides the first systematic evaluation of how different weather types and reasoning dimensions affect VLM performance.

4. Key Results & Findings

Extensive experiments across multiple VLMs reveal two critical findings:

• Monotonic Performance Degradation: VLM performance degrades consistently as weather severity increases.

  • In synthetic benchmarks, average accuracy dropped by ~15% compared to ideal environments.
  • In real-world scenarios, reasoning-based methods achieved only ~50% of the performance upper bound (established by pure perception models like SAM2).
  • Specific Drop: For the Seg-Zero-7B model, gIoU dropped from 65.3 (clean) to 47.7 (severe weather), indicating that visual degradation directly weakens semantic association and spatial grounding.

• Distinct Vulnerability Patterns:

  • Weather Type: Snow causes the most severe degradation (due to loss of texture, contrast, and depth cues), followed by fog and rain. Nighttime conditions also caused significant drops.
  • Reasoning Dimension:
    • Robust: Queries based on Structure and Function (directly observable visual attributes) remained relatively stable.
    • Fragile: Queries based on Application and Requirement (requiring high-level contextual abstraction and scenario-level reasoning) suffered the most significant performance degradation. This suggests that the reasoning stage, rather than low-level feature extraction, is the primary bottleneck.

5. Significance

• Safety Implications: The results highlight a critical vulnerability in deploying VLMs for safety-critical tasks (like autonomous driving) in adverse weather. Current models lack the mechanisms to adapt their reasoning when visual inputs are corrupted.
• Research Direction: The paper argues that future models must incorporate weather-aware reasoning mechanisms capable of modeling uncertainty and adapting semantic grounding under distribution shifts.
• Benchmarking Standard: WeatherReasonSeg sets a new standard for evaluating the robustness of multimodal reasoning, moving beyond "clean room" evaluations to realistic, stress-tested scenarios. It provides a foundation for developing models that can reason effectively despite environmental noise.