Technical Summary: SURGE - Approximation and Training Free Particle Filter for Diffusion Surrogate

Problem Statement

Data assimilation (DA) aims to sequentially estimate the latent state of a dynamical system from noisy, partial observations. While traditional methods rely on explicit mechanistic solvers, modern approaches increasingly utilize diffusion models as "digital twins" to represent complex, high-dimensional system dynamics. These models serve as powerful surrogates for generating probabilistic forecasts.

However, integrating physical observations into diffusion-based forecasting presents a significant challenge. A common practice is to guide the diffusion sampling process using the observation likelihood to steer samples toward observation-consistent states. The paper notes that unless this guidance corresponds to the exact Doob's h-transform (which requires solving an intractable backward Kolmogorov equation in high dimensions), the resulting samples are approximate and introduce systematic bias. Consequently, existing methods often fail to produce samples from the true posterior distribution, particularly in high-dimensional, nonlinear regimes with sparse and noisy data.

The core question addressed is: How can we fuse diffusion-based digital twin simulations with partial observations without introducing systematic bias from approximate guidance?

Methodology: SURGE

The authors propose SURGE (Sequential Unbiased Resampling via Girsanov Estimation), a training-free, inference-time framework that performs data assimilation using particle filtering principles on diffusion trajectories.

Core Concept

Instead of attempting to derive an exact guidance term (which is intractable), SURGE treats the guided diffusion process as a proposal distribution and corrects its bias using Sequential Monte Carlo (SMC) and Girsanov's theorem.

1. Path Measure Formulation: The diffusion surrogate is viewed as defining a path measure over continuous trajectories. The standard guided sampler (using an approximate control potential $G$) generates trajectories from a controlled measure $P_G$.
2. Girsanov Correction: The paper leverages Girsanov's theorem to compute the Radon-Nikodym derivative (likelihood ratio) between the uncontrolled (prior) path measure and the guided path measure. This allows the calculation of importance weights that mathematically correct the bias introduced by the approximate guidance.
3. Sequential Resampling:
   • The algorithm propagates an ensemble of particles (trajectories) through the guided diffusion process.
   • At each step, particles are reweighted based on the observation likelihood and the Girsanov correction term.
   • A resampling step is performed to mitigate particle degeneracy, focusing the ensemble on high-probability regions of the posterior.
4. Progressive Likelihood Incorporation: To stabilize the resampling process and reduce variance in high-dimensional settings, the observation likelihood is incorporated progressively along the internal diffusion time steps (using a telescoping factorization) rather than as a single large update at the endpoint. This ensures particle weights remain balanced.

Algorithmic Flow

• Input: Pretrained diffusion surrogate, noisy observations, initial particle ensemble.
• Process:
  • For each time step $t$, generate $N$ trajectories using the guided SDE (Eq. 4).
  • Compute incremental importance weights using the Girsanov formula (Eq. 5/6), which includes the likelihood reward and the correction for the guidance drift.
  • Normalize weights and perform resampling if the effective sample size (ESS) drops below a threshold.
• Output: An approximation-free ensemble representing the posterior distribution $p(x_{t+1} | y_{1:t+1})$.

Key Contributions

1. Approximation-Free Framework: SURGE is the first work to provide an approximation-free filtering algorithm for diffusion-based surrogates. It rigorously fuses observational data with diffusion simulations without relying on heuristic guidance that biases the posterior.
2. Girsanov-Based Reweighting: The method casts diffusion-based forecasting as inference over a path distribution. It uses a Girsanov change-of-measure to compute importance weights that exactly correct the bias introduced by approximate likelihood guidance.
3. Inference-Time Scaling: SURGE operates solely at the inference phase. It acts as a "plug-and-play" enhancement for existing diffusion-based DA methods (like FlowDAS or SDA) without requiring retraining of the underlying generative model.
4. Stabilization via Progressive Likelihood: The introduction of time-distributed likelihood incorporation mitigates particle degeneracy in high-dimensional, long-horizon tasks.

Experimental Results

The authors evaluated SURGE on three distinct datasets: the Lorenz 1963 system (low-dimensional chaotic), a forced incompressible Navier-Stokes system (high-dimensional fluid dynamics), and the SEVIR weather forecasting dataset (real-world meteorology).

• Lorenz System: SURGE consistently outperformed classical filters (BPF, EnKF) and score-based/flow-based baselines (SDA, FlowDAS) in terms of RMSE and Wasserstein distance ($W_1$), even under extreme partial observation (only the $x$-coordinate observed) and high noise.
• Navier-Stokes: In super-resolution and sparse observation tasks, SURGE achieved lower Root Mean Square Error (RMSE) and better preservation of the kinetic energy spectrum (KES-RE) compared to baselines. Qualitative results showed SURGE maintained structural fidelity and sharp boundaries better than FlowDAS, which tended to produce blurry predictions.
• Weather Forecasting: On the SEVIR dataset (predicting Vertically Integrated Liquid), SURGE improved the Critical Success Index (CSI) and reduced RMSE across various intensity thresholds ($\tau_{20}, \tau_{40}$) compared to FlowDAS and SDA.

In all experiments, SURGE enhanced the performance of the underlying backbone models (SDA and FlowDAS) without additional training.

Significance and Claims

The paper claims that SURGE provides a rigorous, approximation-free solution to the problem of data assimilation with diffusion surrogates. By decoupling the generation of proposals (via the diffusion model) from the correction of bias (via Girsanov-weighted resampling), the method avoids the systematic errors inherent in approximate guidance.

The authors emphasize that:

• SURGE is training-free, making it a universal online enhancement for existing diffusion-based sequence models.
• It is particularly effective in high-dimensional, nonlinear, and chaotic systems where traditional particle filters suffer from weight degeneracy and where exact guidance is intractable.
• While the method is robust, its efficacy is intrinsically linked to the quality of the base diffusion surrogate; if the surrogate's proposal distribution is fundamentally unstable (e.g., near null-isoclines in the Lorenz system), the resampling mechanism may struggle, highlighting a dependency on the underlying model's stability.

The work positions SURGE as a bridge between the expressive power of generative diffusion models and the rigorous statistical requirements of data assimilation, enabling physically consistent state estimation without the computational cost of retraining or solving intractable backward equations.