WeatherCity: Urban Scene Reconstruction with Controllable Multi-Weather Transformation

Imagine you are a director filming a movie about a busy city street. You have a perfect camera recording the scene on a sunny day. But now, you need to show what that same street looks like during a heavy downpour, a blizzard, or a thick fog.

In the past, trying to do this was like trying to paint a new sky onto a photograph. If you just used a photo editor, the rain might look fake, the cars might warp into weird shapes, or the puddles might appear in the wrong places. It was like trying to change the weather in a painting by smudging the paint—you lose the details of the original picture.

WeatherCity is a new "magic tool" that solves this problem. Instead of just editing a flat picture, it builds a 3D digital twin of the city first, and then lets you change the weather inside that 3D world.

Here is how it works, broken down into simple concepts:

1. The "Lego City" vs. The "Painting"

Most old methods treated the video like a 2D painting. If you wanted to add snow, you just painted white over the cars. The problem? The snow didn't stick to the car's shape; it just sat on top, and if the car moved, the snow looked like it was sliding off a flat sheet of glass.

WeatherCity builds the city out of millions of tiny, invisible "digital Lego bricks" (called Gaussians).

The Structure: These bricks know exactly where the road, the buildings, and the cars are in 3D space. They hold the "skeleton" of the city.
The Skin: The tool separates the shape of the city from the look of the weather. Think of it like a mannequin (the city structure) and a wardrobe (the weather). You can swap the wardrobe from "Sunny" to "Snowy" without ever changing the shape of the mannequin.

2. The "Smart Translator" (Text-Guided Editing)

You don't need to be a graphic designer. You just type a sentence, like "Make it rain heavily on this street."

The system uses a powerful AI (like a super-smart translator) to understand your words.
It looks at your original video and says, "Okay, I see the street. I will keep the buildings and cars exactly where they are, but I will change the lighting and atmosphere to match 'heavy rain'."
This ensures that if a car was red in the original video, it stays red in the rain. It doesn't accidentally turn the car blue or make it disappear.

3. The "Physics Engine" (Real Rain and Snow)

This is where WeatherCity gets really cool. Other tools might just paste a static image of rain over the video. But real rain moves!

The Particles: WeatherCity creates thousands of tiny, invisible "digital raindrops" and "snowflakes."
The Physics: It gives them rules. Rain falls straight down but gets pushed sideways by the wind. Snowflakes flutter and drift.
The Result: Because these particles are part of the 3D world, they interact correctly. If a raindrop falls behind a car, the car blocks it. If it falls in front, you see it clearly. It creates a realistic "depth" that flat editing can't do.

4. The "Foggy Window" (Fog Simulation)

Fog is tricky because it gets thicker the further away you look.

WeatherCity uses a scientific rule (called the Beer-Lambert law) to simulate this.
It calculates how much light gets blocked as it travels through the "digital fog."
The result? Distant buildings fade away naturally, while the car right in front of the camera stays clear. It looks exactly like looking through a real foggy window.

Why Does This Matter?

Imagine you are training a self-driving car. You want to teach it how to drive in a blizzard, but you can't wait for a blizzard to happen in real life, and it's dangerous to test on real roads.

Before: You had to hope the weather would cooperate, or use fake, low-quality simulations that didn't look real.
With WeatherCity: You can take a video of a sunny day and instantly turn it into a dangerous blizzard or a heavy fog. You can test the car's AI in thousands of different weather scenarios without ever leaving the lab.

In a Nutshell

WeatherCity is like having a time-traveling weather machine for video. It takes a real city, builds a 3D model of it, and lets you dial up the rain, snow, or fog with a text message. It keeps the city looking real and consistent, while adding physics-based weather that moves and interacts with the world just like the real thing.

1. Problem Statement

High-fidelity 4D urban scene reconstruction is critical for autonomous driving, particularly for end-to-end training and closed-loop simulation in diverse weather conditions. However, existing methods face three primary limitations:

Observation Dependency: Current 4D reconstruction techniques (e.g., NeRF, 3D Gaussian Splatting) can only reproduce the specific weather conditions present during data capture. They cannot simulate challenging scenarios like rain, snow, or fog.
Inconsistency in 2D Editing: Image-level weather editing methods (GANs, Diffusion models) often introduce scene artifacts, hallucinations (e.g., distorted buildings or road markings), and temporal flickering. They lack fine-grained control over weather intensity and fail to maintain geometric consistency across frames.
Lack of Dynamic Simulation: Existing 3D weather editing approaches are often limited to static effects (e.g., snow cover) or removing artifacts (deraining), failing to simulate dynamic phenomena like falling rain or drifting snow with physical realism.

2. Methodology

WeatherCity proposes a unified framework that integrates 4D Reconstruction, Controllable Weather Editing, and Dynamic Simulation. The framework consists of four core modules:

A. Text-Guided Image Weather Background Editing

Mechanism: Utilizes a powerful text-guided image editing foundation model (Qwen-Image) to generate target weather sequences from raw input images.
Strategy: Employs meticulously designed text prompts that explicitly enforce the strict preservation of original scene content (layout, objects, geometry) while applying the desired weather effect (e.g., "rainy city street").
Role: These edited sequences serve as supervision signals for the 3D reconstruction phase.

B. Weather Gaussian Representation (Core Innovation)

To ensure structural consistency across different weather conditions, the authors introduce a novel representation that disentangles scene geometry from weather appearance:

Shared Scene Features: Each 3D Gaussian primitive ( $G_i$ ) in the scene graph (background, rigid objects, non-rigid objects) is parameterized by a shared appearance feature vector ( $f_i$ ). This vector encodes intrinsic texture and material properties independent of weather.
Multi-Weather Decoders: A set of weather-specific Multi-Layer Perceptrons (MLPs), denoted as $\phi_w$ , decode the shared features into specific RGB colors ( $c_i^w$ ) for each weather condition $w$ .
Result: This architecture allows the underlying scene geometry to remain stable while enabling flexible switching between weather states (e.g., sunny to rainy) without altering object shapes or positions.

C. Multi-Weather Consistency Optimization

The system optimizes the Gaussian representation using a composite loss function to align rendered outputs with both raw and edited images:

RGB Loss: Minimizes pixel-wise and structural (SSIM) differences between rendered and ground-truth images.
Content Consistency Loss ( $L_{cc}$ ): Uses a pre-trained VGG network to extract features. It enforces that the semantic content of the rendered image under weather $w$ matches the original clear-weather image, preventing the 2D editing model from introducing geometric distortions.
Depth Loss: Supervises geometry using sparse LiDAR depth maps.

D. Physics-Driven Dynamic Weather Simulation

To achieve realistic dynamic effects, the framework simulates weather particles and atmospheric conditions physically:

Rain & Snow Particles: Modeled as Gaussian ellipsoids.
- Rain: Elongated Gaussians aligned with velocity vectors to simulate motion blur.
- Snow: Three concentric Gaussians forming a crystal shape, with stochastic turbulence added for fluttering motion.
- Motion: Particles follow physics-driven velocity equations incorporating gravity, wind vectors, and turbulence.
Fog Rendering: Implemented using the Beer-Lambert law for depth-aware attenuation, ensuring fog density increases naturally with distance.
Unified Rendering: Dynamic weather particles are integrated directly into the Gaussian scene graph as dedicated nodes ( $N_{rain}, N_{snow}$ ), allowing for correct occlusion and blending with the scene during standard Gaussian Splatting rasterization.

3. Key Contributions

Unified Framework: The first system to integrate 4D reconstruction, flexible weather editing, and dynamic simulation, elevating 2D editing to 4D simulation for autonomous driving.
Weather Gaussian Representation: A novel representation using shared features and multi-weather decoders that successfully disentangles scene structure from weather appearance, ensuring geometric consistency across conditions.
Physics-Driven Simulation: A dynamic simulation system using particle physics and optical laws (Beer-Lambert) to generate controllable, realistic, and temporally consistent dynamic weather effects (rain, snow, fog).
Fine-Grained Control: Enables precise manipulation of weather intensity (e.g., light vs. heavy rain) and object-level editing (e.g., removing specific vehicles) within the reconstructed scene.

4. Experimental Results

The method was evaluated on the Waymo Open Dataset and nuScenes dataset, comparing against state-of-the-art image editing (ControlNet, TurboEdit), video editing (FRESCO), and 3D editing (ClimateNeRF) baselines.

Quantitative Performance: WeatherCity achieved State-of-the-Art (SOTA) results across all metrics:
- CLIP-S (Content Preservation): ~0.87–0.89 (vs. ~0.63–0.83 for baselines).
- CLIP-DS (Prompt Alignment): ~0.30 (vs. ~0.21–0.28 for baselines).
- Sem-CS (Semantic Consistency): ~0.91–0.97 (vs. ~0.69–0.90 for baselines).
Qualitative Results:
- Consistency: Unlike baselines which caused warped vehicles and hallucinated structures, WeatherCity preserved scene geometry and semantic layout perfectly.
- Dynamic Effects: Successfully rendered realistic falling rain, snow accumulation, and depth-attenuated fog, which 2D methods failed to do.
- Temporal Coherence: Eliminated inter-frame flickering common in video editing baselines.
Efficiency: Achieved a rendering speed of 25.67 FPS, meeting real-time simulation requirements, whereas baselines operated at <0.15 FPS.

5. Significance

WeatherCity addresses a critical gap in autonomous driving simulation by providing a tool that can generate diverse, edge-case weather scenarios (rain, snow, fog) with high fidelity and geometric consistency.

Safety & Training: It enables the creation of reproducible virtual testing environments for extreme weather, crucial for training robust end-to-end driving models.
Versatility: Beyond weather, the framework supports object-level manipulation (adding/removing cars/pedestrians), making it a comprehensive tool for synthetic data generation.
Future Impact: It establishes a foundation for creating controllable, dynamic virtual environments, moving beyond static scene reconstruction to fully interactive 4D world modeling.