IntrinsicWeather: Controllable Weather Editing in Intrinsic Space

IntrinsicWeather is a diffusion-based framework that enables controllable weather editing by decomposing images into intrinsic maps for enhanced spatial control and leveraging CLIP-space interpolation for fine-grained weather generation, outperforming existing methods and benefiting downstream tasks like autonomous driving.

The Gist

Imagine you have a photograph of a street on a sunny day. Now, imagine you want to turn that photo into a rainy, snowy, or foggy scene.

Most current AI tools try to do this by painting over the pixels, like a digital artist smudging colors on a canvas. The problem? They often mess up the underlying scene. They might make a car look like a puddle, change the shape of a building, or make the shadows look weird because they are just guessing how the weather affects the image, not the world inside the image.

IntrinsicWeather is a new AI framework that takes a smarter, more "physics-based" approach. Instead of painting over the photo, it acts like a digital stage manager who breaks the scene down into its fundamental parts, swaps out the weather, and rebuilds the scene perfectly.

Here is how it works, using a simple analogy:

1. The "Deconstruction" Phase (The Inverse Renderer)

Imagine you have a complex Lego castle. To change the weather, you don't just spray paint the whole thing. Instead, you take it apart into three specific piles:

• The Bricks (Materials): The color and texture of the road, the car, and the trees. These don't change whether it's raining or sunny.
• The Shape (Geometry): The 3D structure of the scene. The road is still flat; the car is still a boxy shape.
• The Lighting (Irradiance): This is the "weather" part. It includes the sun, the rain, the fog, and the shadows.

IntrinsicWeather uses a special AI "decoder" to look at your photo and separate these three piles. It strips away the rain or snow to reveal the clean, dry "bricks and shape" underneath. This is called Inverse Rendering.

2. The "Reconstruction" Phase (The Forward Renderer)

Now that you have the clean piles of bricks and shape, you want to build a new scene.

• You keep the Bricks and Shape exactly the same (so the car doesn't turn into a tree).
• You grab a new Lighting set based on a text prompt you give the AI, like "Make it a heavy snowstorm."
• The AI "re-builds" the image, applying the new snow and shadows onto the original, unchanged structure.

This is called Forward Rendering. Because the AI knows the difference between the object and the weather, the result looks incredibly realistic. The snow piles up naturally on the car, and the wet road reflects the streetlights correctly.

The Secret Sauce: "The Spotlight" (IMAA)

One of the biggest challenges is that outdoor scenes are huge. A car far away is tiny, while a building nearby is huge. Standard AI often gets confused and misses the small details.

The authors added a clever trick called Intrinsic Map-Aware Attention. Think of this as giving the AI a flashlight.

• When the AI needs to figure out the shape of a road, the flashlight shines on the road.
• When it needs to figure out the metal on a car, the flashlight shines on the car.
• This ensures the AI pays attention to the right details in the right places, preventing it from hallucinating weird shapes or missing small objects like distant pedestrians.

Why Does This Matter?

You might ask, "Why not just use a filter?"

• For Self-Driving Cars: Self-driving cars need to "see" clearly. If it's raining, their cameras get confused. IntrinsicWeather can take a rainy photo, strip away the rain to see the road clearly, and then test if the car's safety software works. It helps train robots to drive in bad weather without needing to wait for a real storm.
• For Creativity: It allows photographers and filmmakers to change the weather in a scene without ruining the physics of the image. You can turn a sunny day into a blizzard, and the snow will land exactly where it should.

The "Training Gym"

To teach this AI how to do all this, the researchers built two massive new "training gyms" (datasets):

1. WeatherSynthetic: 38,000 computer-generated scenes with perfect weather data.
2. WeatherReal: 18,000 real-world photos where they used their own AI to guess the weather-free version.

In a Nutshell

IntrinsicWeather is like a time-traveling photographer who can strip a scene down to its bare bones, swap the weather like changing a costume, and put it back together so perfectly that the physics, lighting, and geometry remain flawless. It moves beyond simple "photo editing" into "scene reconstruction," making it a powerful tool for both safety technology and creative art.

Technical Summary

Here is a detailed technical summary of the paper "IntrinsicWeather: Controllable Weather Editing in Intrinsic Space".

1. Problem Statement

Current weather editing methods for autonomous driving and outdoor scenes face significant limitations:

• Pixel-Space Limitations: Existing diffusion-based editing tools (e.g., Instruct-Pix2Pix, WeatherWeaver) operate directly on pixel values. This entangles weather effects with underlying scene geometry and material properties, leading to structural inconsistencies, hallucinated objects, and unnatural lighting when changing weather conditions.
• Lack of Physical Interpretability: Pixel-space methods struggle to preserve the scene's intrinsic structure (albedo, roughness, geometry) while modifying illumination and weather particles.
• Generalization Gap: Prior intrinsic decomposition methods (e.g., RGB↔X, DiffusionRenderer) are primarily designed for indoor scenes or small objects and fail to generalize to large-scale, unconstrained outdoor driving environments with diverse weather conditions.
• Data Scarcity: There is a lack of large-scale datasets containing paired outdoor images and high-quality intrinsic map annotations (albedo, normal, roughness, etc.) under various weather conditions.

2. Methodology

The authors propose IntrinsicWeather, a diffusion-based framework that operates in intrinsic space rather than pixel space. The framework consists of two main components:

A. Weather-Aware Inverse Renderer

This component decomposes a single input image into weather-invariant and weather-variant intrinsic maps:

• Decomposition Targets:
  • Weather-Invariant: Albedo, Roughness, Metallicity, and Normal maps (representing material and geometry).
  • Weather-Variant: Irradiance map (capturing illumination and weather effects like fog or rain).
• Intrinsic Map-Aware Attention (IMAA): To address the challenge of varying object scales in outdoor scenes, the authors introduce a novel attention mechanism.
  • It uses DINOv2 to extract patch tokens.
  • Learnable embeddings for each intrinsic map type act as a gating mechanism to filter patch tokens, creating an attention bias.
  • This forces the diffusion model to focus on semantically relevant regions (e.g., focusing on geometry details for normal maps, or metallic objects for metallicity maps), significantly improving decomposition fidelity.

B. Weather-Conditioned Forward Renderer

This component synthesizes a new image based on the decomposed intrinsic maps and a text prompt describing the target weather:

• CLIP-Space Interpolation: To achieve fine-grained control over weather intensity (e.g., light rain vs. heavy rain), the method interpolates between text embeddings in CLIP space.
  • A weather transition direction vector $e$ is calculated between the target weather ($w_1$) and original weather ($w_2$).
  • The prompt embedding is shifted along this direction by a factor $\alpha$ to generate intermediate weather states.
• Feature Distillation: The forward renderer aligns its intermediate features with the pre-trained Stable Diffusion 3.5 to preserve the model's strong generative priors while adapting to the specific weather conditions.

C. Dataset Construction

To train these components, the authors created two new datasets:

1. WeatherSynthetic: A large-scale synthetic dataset (38k images) generated using Unreal Engine 5. It covers diverse weather (sunny, rainy, snowy, foggy, sandstorm), times of day, and environments (urban, highway). It provides ground-truth intrinsic maps.
2. WeatherReal: A real-world dataset (18k images) derived from open-source datasets (Waymo, KITTI). It uses the inverse renderer to generate pseudo-labels, refined by multi-modal models and manual checks, to bridge the sim-to-real gap.

3. Key Contributions

1. IntrinsicWeather Framework: A novel two-stage pipeline (Inverse + Forward rendering) that enables controllable weather editing by disentangling materials, geometry, and illumination.
2. Intrinsic Map-Aware Attention (IMAA): A mechanism that enhances spatial correspondence and decomposition quality in complex outdoor scenes by guiding the diffusion model to focus on specific regions relevant to each intrinsic map.
3. New Datasets: The release of WeatherSynthetic and WeatherReal, the first large-scale datasets providing paired outdoor driving images and intrinsic map annotations under diverse weather conditions.
4. Fine-Grained Control: The ability to generate natural transitional weather states (e.g., gradual accumulation of snow or wetness) via CLIP-space interpolation, surpassing binary "remove/add" approaches.

4. Experimental Results

The method was evaluated against state-of-the-art pixel-space editing, weather restoration, and rendering-based methods.

• Quantitative Performance:
  • Inverse Rendering: Achieved a >10 dB PSNR improvement over existing methods (e.g., 27.99 PSNR for Albedo vs. ~11-16 for others) on the WeatherSynthetic test set.
  • Forward Rendering: Achieved the highest PickScore and CLIP-S (text-image consistency) among rendering-based methods, indicating superior alignment with text prompts and physical plausibility.
• Qualitative Performance:
  • Successfully removed weather artifacts (snow, rain streaks, fog) while preserving scene geometry and material consistency.
  • Pixel-space methods often distorted car colors, changed pedestrian poses, or failed to remove ground-level snow.
• Downstream Task Improvement:
  • Applying IntrinsicWeather to the ACDC benchmark significantly boosted autonomous driving perception models:
    • Object Detection (DETR): AP75 increased from 13.15% to 24.60% (+87.1% relative gain).
    • Semantic Segmentation (Segformer): mIOU increased from 24.13% to 30.05% (+24.5% relative gain).
• User Study: Users preferred the results of IntrinsicWeather over competitors in 81.67% of cases, citing better realism and consistency.

5. Significance

• Robustness for Autonomous Driving: By correcting environmental distortions at the visual input level, the framework significantly enhances the robustness of perception models (detection and segmentation) in adverse weather, a critical safety requirement for self-driving vehicles.
• Physical Plausibility: Moving editing from pixel space to intrinsic space ensures that generated images adhere to physical laws of lighting and material, preventing the "hallucinations" common in generative editing.
• Data-Driven Advancement: The release of the new datasets addresses a critical bottleneck in the field, enabling future research in weather-invariant scene understanding and rendering.
• Limitations: The current framework is designed for single-image editing and does not guarantee temporal consistency in video sequences, though the authors suggest this could be extended using video diffusion priors in future work.