AutoAWG: Adverse Weather Generation with Adaptive Multi-Controls for Automotive Videos

1. Problem Statement

Autonomous driving systems require robust perception under adverse weather conditions (rain, snow, fog, night). However, the core bottleneck is the extreme scarcity of real-world video data capturing these specific conditions.

• Existing Limitations:
  • Weather Removal: Difficult to deploy in real-time and does not generate new data for training.
  • Weather Generation (Style Transfer): Existing methods struggle to balance visual realism (strong weather stylization) with semantic-structural integrity (preserving the geometry and annotations of safety-critical objects like vehicles and pedestrians).
  • Data Scarcity: Acquiring paired data (same scene, different weather) is practically impossible. Synthetic data introduces domain gaps and artifacts.
  • Annotation Reusability: Many generation methods alter scene geometry, rendering existing 2D/3D annotations unusable for downstream perception tasks.

2. Methodology: AutoAWG Framework

AutoAWG is a controllable video diffusion framework designed to synthesize adverse weather while preserving scene structure. It treats video weather transfer as a "coloring-book" process where structural priors guide the generation of weather textures.

A. Adaptive Multi-Control Fusion

The core innovation is a semantics-guided adaptive fusion of multiple control maps to balance style and content:

• Control Inputs:
  • Lineart: Captures fine-grained details and boundaries (high content fidelity, low style flexibility).
  • Sketch: Captures global structure and layering (balanced).
  • Depth: Captures coarse structural layout (high style flexibility, low detail).
• Fusion Strategy: Instead of using all controls uniformly, the model uses semantic masks to selectively apply them:
  • Safety-Critical Objects (Vehicles, Pedestrians, Signs): Only the Lineart map is retained to strictly preserve boundaries and geometry.
  • Sky/Background: The Sketch map is masked out to allow flexible weather variations (e.g., changing sky color for night/fog).
  • Global Structure: The Depth map is preserved entirely.
• Formula: $C_a = \text{concat}\{C_d, C_l \odot M_{obj}, C_s \odot (1 - M_{sky})\}$, where $M$ represents masks.

B. Importance-Weighted Loss

To ensure the model prioritizes safety-critical regions during training, an importance-weighted loss is introduced within the Flow Matching framework:
$$L = E[\|U_t - V_t\|^2 + \alpha \cdot \|M_{obj} \odot (U_t - V_t)\|^2]$$
This forces the model to minimize reconstruction errors specifically for objects (vehicles, pedestrians), ensuring their annotations remain valid.

C. Data Synthesis: VP-Anchored Temporal Synthesis

To address the lack of adverse weather video data, the authors propose a method to convert static images (from the ACDC dataset) into pseudo-videos:

1. Vanishing Point Anchoring: The vanishing point of the image is estimated.
2. Cropping: The image is cropped with a fixed aspect ratio, ensuring the vanishing point's relative position remains constant.
3. Synthesis: Intermediate crops are resized and concatenated along the temporal dimension to simulate forward driving motion.
   This creates a "pseudo-video" from static images, reducing reliance on synthetic data pipelines.

D. Long-Horizon and Multi-View Generation

• Masked Segmented Training: To support indefinite video lengths, the model is trained using a masking strategy where either all frames are masked (for the first segment) or all but the first frame are masked (for continuation). This compels the model to learn long-range temporal dependencies.
• Multi-Camera Support: Frames from multiple cameras are stitched into a single composite grid. The DiT (Diffusion Transformer) models relationships across these tokens via self-attention, ensuring cross-view consistency without complex explicit cross-view attention mechanisms.

3. Key Contributions

1. Adaptive Multi-Control Mechanism: A novel "coloring-book" approach that decouples style (weather) from content (structure) by adaptively fusing depth, sketch, and lineart maps based on semantic regions.
2. VP-Anchored Temporal Synthesis: A strategy to generate training sequences from static images, effectively mitigating data scarcity for adverse weather without introducing heavy synthetic domain gaps.
3. Annotation Reusability: By strictly preserving object geometry via importance-weighted loss and selective control fusion, the generated videos allow for the direct reuse of existing 2D/3D annotations, enabling plug-and-play data augmentation.
4. Scalability: The framework natively supports multi-camera setups and arbitrary-length video generation.

4. Experimental Results

The method was evaluated on the nuScenes validation set and compared against state-of-the-art (SOTA) video generation models (e.g., MagicDrive, Vista, Panacea).

• Quantitative Performance:

  • FID (Fréchet Inception Distance): Reduced by 50.0% (without first-frame conditioning) and 8.7% (with first-frame conditioning) compared to SOTA.
  • FVD (Fréchet Video Distance): Reduced by 16.1% (without conditioning) and 7.2% (with conditioning).
  • Downstream Utility: When used to augment training data for a BEVFusion 3D object detector, the model improved mAP by 1.99 and NDS by 1.36.
  • Edit Fidelity: The mAP drop for generated frames was significantly lower (0.0177) compared to MagicDrive-V2 (0.1736), proving better preservation of scene semantics.

• Qualitative Performance:

  • Generated videos exhibit high temporal consistency and realistic weather effects (fog, rain, snow, night).
  • Safety-critical objects (cars, pedestrians) remain structurally intact and consistent across frames and camera views.
  • The model successfully generalizes to unseen datasets (Cityscapes, BDD100K) and synthesizes weather types (fog/snow) not present in the nuScenes training set.

5. Significance

AutoAWG addresses a critical gap in autonomous driving research by providing a low-cost, high-fidelity data augmentation solution.

• Practical Value: It solves the "annotation reusability" problem, allowing researchers to generate massive amounts of adverse weather data without the prohibitive cost of re-labeling.
• Robustness: The framework's ability to maintain geometric consistency ensures that the generated data is safe and effective for training perception models, directly contributing to the safety of autonomous vehicles.
• Scalability: The ability to handle multi-camera systems and long sequences makes it suitable for real-world simulation and testing environments.

The code is open-sourced at: https://github.com/higherhu/AutoAWG.