Pretrained Video Models as Differentiable Physics Simulators for Urban Wind Flows

1. Problem Statement

Designing safe and comfortable urban spaces requires accurate, time-resolved Computational Fluid Dynamics (CFD) simulations to predict pedestrian wind comfort and safety. However, traditional CFD methods face two critical limitations:

• Computational Cost: High-fidelity simulations are slow (minutes to hours), making extensive design exploration and optimization impractical.
• Non-Differentiability: Standard CFD solvers are not differentiable, meaning they cannot provide gradient signals to guide the geometric optimization of building layouts.

Existing deep learning surrogates (e.g., CNNs, Graph Neural Networks, Fourier Neural Operators) often fail to capture transient flow phenomena (like vortex shedding) required for wind comfort criteria, which are defined as exceedance probabilities over threshold speeds. Furthermore, autoregressive models often suffer from error accumulation over long time rollouts.

2. Methodology

The authors introduce WinDiNet (Wind Diffusion Network), a framework that repurposes a pretrained latent video diffusion model as a fast, differentiable physics simulator.

A. Data Generation

• Dataset: 13,000 2D incompressible CFD simulations generated using an Euler solver.
• Scenarios: Procedurally generated urban layouts with varying building configurations, inlet speeds ($u_{in} \in [0.1, 20]$ m/s), and domain sizes ($L \in [900, 1400]$ m).
• Format: Simulations are 112 frames long at $256 \times 256$ resolution.
• Encoding: Physical velocity fields $(u, v)$ and building footprints are encoded into RGB channels:
  • Red: Horizontal velocity ($u$).
  • Green: Vertical velocity ($v$).
  • Blue: Fluid mask (1 for fluid, 0 for building).
  • Conditioning: A "frame 0" encodes the inlet speed and geometry to guide the generation.

B. Model Architecture

The base model is LTX-Video, a 2B-parameter latent video diffusion transformer (DiT).

• Training Strategy: The model is fine-tuned on the urban wind dataset.
• Conditioning Mechanism: Instead of text prompts (which are imprecise for continuous physics), the authors use scalar conditioning. Inlet speed and domain size are normalized, encoded via Fourier features, and projected into embedding tokens to replace the text encoder output in the cross-attention layers.
• VAE Adaptation: Since the pretrained VAE is designed for natural RGB video, it introduces artifacts when decoding pseudo-colored velocity fields. The authors propose three adaptation strategies:
  1. Color Adapter: A learnable MLP mapping physical channels to a new color space before the encoder and back after the decoder.
  2. Decoder Fine-Tuning (Dec. FT): Fine-tuning only the VAE decoder weights while keeping the encoder frozen.
  3. Physics-Informed Loss: Adding physical constraints (divergence-free, no-penetration at walls, distance-weighted MSE) during the decoder fine-tuning stage.

C. Inverse Optimization

Because the surrogate is end-to-end differentiable, it enables gradient-based inverse optimization:

1. Differentiable Rasterization: Building coordinates are mapped to occupancy masks using a soft rasterizer (sigmoid-based union of rectangles), allowing gradients to flow from the simulation output back to building positions.
2. Objective Function: A composite loss penalizes wind speeds outside target comfort bands (e.g., $>15$ m/s is dangerous, $<1$ m/s is stagnant).
3. Modes:
   • Rigid: Buildings translate as whole units.
   • Morph: Buildings are subdivided into independent sub-blocks that can deform, allowing for shape optimization.

3. Key Contributions

1. Dataset Creation: A large-scale dataset of 13,000 procedurally generated 2D urban wind simulations.
2. Systematic Study: A comprehensive analysis of adapting pretrained video models for physics, comparing training regimes (LoRA vs. Full Fine-tuning), conditioning (Text vs. Scalar), and VAE adaptation strategies.
3. Performance Benchmarking: Demonstration that a fine-tuned video diffusion model outperforms purpose-built neural PDE solvers (like FNO, RNO, OFormer) in accuracy and temporal coherence.
4. Differentiable Design Workflow: The first demonstration of using a video diffusion model as a differentiable physics simulator for gradient-based urban layout optimization, achieving results in under a second.

4. Key Results

• Accuracy: The best variant (Dec. FT Physics) achieved a Variance-Normalized RMSE (VRMSE) of 0.520, outperforming the best baseline neural operator (RNO at 0.563) by 7.6%. It also achieved the lowest Mean Absolute Error (MAE) of 1.01 m/s.
• Speed: WinDiNet generates full 112-frame rollouts in ~0.32 seconds on a single GPU, roughly three orders of magnitude faster than the CFD solver used for training.
• VAE Adaptation: Decoder fine-tuning with physics-informed losses proved critical, reducing reconstruction VRMSE by over 62% compared to the frozen baseline.
• Optimization Efficacy:
  • In single-inlet scenarios, the optimizer reduced dangerous wind speeds ($>15$ m/s) from 2.6% to 0.2% and uncomfortable speeds ($>5$ m/s) from ~50% to ~13%.
  • In multi-inlet scenarios (conflicting wind directions), the model successfully found compromise geometries that satisfied comfort constraints for both directions simultaneously.
• Generalization: The model generalized well to real-world city layouts (Barcelona, Berlin, Paris, Zurich) despite being trained only on synthetic data. It also showed graceful degradation when extrapolating to inlet speeds and domain sizes slightly outside the training distribution.

5. Significance and Impact

• Paradigm Shift: This work challenges the notion that video generation models are merely perceptual tools. It proves that foundation models pretrained on natural video can learn quantitative physical laws (Navier-Stokes dynamics) when fine-tuned on specific data.
• Design Workflow: By combining speed and differentiability, WinDiNet enables a real-time design loop where architects can iteratively adjust building layouts and immediately see the impact on wind safety, a task previously impossible with traditional CFD.
• Future Directions: The authors suggest this approach can extend to 3D simulations and other physical domains. The use of non-autoregressive diffusion models avoids the error accumulation issues common in autoregressive PDE solvers, making them ideal for long-horizon physical predictions.

In summary, WinDiNet bridges the gap between generative AI and scientific computing, offering a powerful, differentiable, and ultra-fast surrogate for urban wind engineering that outperforms specialized neural operators while enabling novel inverse design capabilities.