📊 statistics

On the Wasserstein Gradient Flow Interpretation of Drifting Models

This paper analyzes the Generative Modeling via Drifting (GMD) framework by Deng et al. (2026) through the lens of Wasserstein Gradient Flows, demonstrating that the proposed algorithm targets a fixed point of a KL-divergence flow while the actual implementation resembles a Sinkhorn divergence flow with limitations, and further extending this perspective to other divergences like MMD and sliced Wasserstein distance.

Original authors: Arthur Gretton, Li Kevin Wenliang, Alexandre Galashov, James Thornton, Valentin De Bortoli, Arnaud Doucet

Published 2026-05-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Arthur Gretton, Li Kevin Wenliang, Alexandre Galashov, James Thornton, Valentin De Bortoli, Arnaud Doucet

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to draw a picture of a cat. The robot starts by scribbling random lines (noise). Your goal is to nudge those lines until they look exactly like a real cat photo.

This paper analyzes a new method called "Generative Modeling via Drifting" (GMD). Think of "drifting" as a gentle, continuous push that moves the robot's random scribbles closer to the real cat photo, step by step.

The authors, researchers from Google DeepMind, wanted to understand why this method works and what is actually happening under the hood. They used a mathematical framework called Wasserstein Gradient Flows (WGF).

Here is the simple breakdown of their findings:

1. The Big Idea: The "Steepest Descent" Path

Imagine you are standing on a foggy mountain (the "loss landscape") and you want to get to the valley floor (the perfect cat picture).

Wasserstein Gradient Flow is like a GPS that tells you the exact steepest path down the mountain. It doesn't just say "go down"; it calculates the most efficient route based on how the "shape" of the mountain changes.
The paper argues that the GMD method is essentially trying to find the very bottom of this valley (the "fixed point") where the robot's drawing matches the real photo perfectly.

2. The Two Versions of the Algorithm

The authors found that the GMD method has two faces, and they behave differently:

Face A: The "Score Difference" (The Simple Version)

The Analogy: Imagine you have a map of where real cats are (the target) and a map of where your robot's drawings are (the model).
How it works: The algorithm calculates the "score" (a measure of density) for both maps. It then pushes the robot's drawing in the direction that reduces the difference between the two maps.
The Finding: The authors showed that this simple version is mathematically equivalent to trying to find the bottom of a valley defined by KL Divergence (a standard way to measure how different two probability distributions are). It's like smoothing out the maps with a blur filter (Parzen smoothing) before comparing them.

Face B: The "Sinkhorn Proxy" (The Real Version)

The Reality: The actual code the researchers used in practice (Algorithm 2) is more complex. It doesn't just compare smoothed maps; it tries to solve a specific transportation puzzle called the Sinkhorn divergence.
The Analogy: Imagine you have a pile of sand (your robot's drawings) and a mold (the real cat). You want to move the sand to fill the mold with the least amount of effort. The "Sinkhorn" method is a clever, fast way to figure out exactly which grain of sand goes where.
The Finding: The authors proved that the real algorithm acts like a "proxy" (a stand-in) for this optimal transport flow.
- Good News: If the robot's drawing matches the real cat perfectly, the "push" stops (velocity becomes zero).
- Bad News: The authors discovered a flaw. Even though it looks like an optimal transport method, it doesn't actually behave like one in all situations. Specifically, if the robot is trying to move a pile of sand from one side of a wide canyon to the other, this method might get stuck or move too slowly because the "bridge" (the kernel) is too narrow. It fails to move mass across large gaps as efficiently as a true optimal transport method should.

3. What Else Can This Do?

The paper suggests that this "drifting" idea isn't limited to just one type of math. You can apply this same "pushing" logic to other ways of measuring differences:

MMD (Maximum Mean Discrepancy): Another way to compare distributions.
Sliced Wasserstein: A method that compares distributions by looking at them from different angles (like slicing a loaf of bread).
GAN Critics: The "judge" networks used in Generative Adversarial Networks.

The authors show that you can build a "drifting" generator for any of these methods, not just the one originally proposed.

Summary of the "Verdict"

What it is: A new way to train AI to generate data by gently "drifting" samples toward the truth.
What it is (Mathematically): It's an attempt to find the bottom of a valley defined by optimal transport distances.
The Catch: The specific algorithm used in practice is a clever shortcut (a "proxy") for a complex math problem. While it works well, the authors proved it has some theoretical limitations when the data is very spread out (like trying to move sand across a wide canyon). It behaves more like a standard smoothing method in those tough cases, rather than the perfect "optimal transport" method it claims to mimic.

In short: The paper takes a new, successful AI trick, explains the math behind it, identifies exactly what it is doing, and points out where the math gets a little shaky.

Technical Summary: On the Wasserstein Gradient Flow Interpretation of Drifting Models

Problem Statement

Generative Modeling via Drifting (GMD), proposed by Deng et al. (2026), is a novel framework for generative tasks that achieves strong empirical performance without adversarial training, distillation, or multi-step sampling. Instead, it relies on a non-parametric, mean-shift style drift operator to iteratively move generated samples toward a target data distribution. Despite its practical success, the theoretical underpinnings of GMD remain under-explored. Specifically, there is a lack of rigorous analysis regarding how the specific algorithms proposed by Deng et al. relate to the geometry of optimal transport and Wasserstein Gradient Flows (WGF).

Methodology

This work analyzes GMD through the lens of Wasserstein Gradient Flows, which describe the path of steepest descent for a functional in the space of probability measures equipped with the geometry of optimal transport. The authors distinguish between two primary instantiations of the drifting concept:

Theoretical Instantiation (Score-Difference Drift): The authors first analyze a simpler version of GMD (Algorithm 1 in Deng et al.) where the drift field is defined as the difference between attraction (from target data) and repulsion (from model samples) terms. They demonstrate that this specific drift field corresponds to the difference of scores of Parzen-smoothed densities.
Practical Instantiation (The Sinkhorn Proxy): The authors focus on the algorithm actually implemented by Deng et al. (Algorithm 2), which is more complex. They propose a theoretical construct called the "Sinkhorn Proxy," which approximates the velocity field of a Wasserstein Gradient Flow on the debiased Sinkhorn divergence. This proxy is derived by replacing the exact, iterative Sinkhorn solution with a one-shot approximation involving geometric means of row-wise and column-wise softmax normalizations, followed by a cross-weighting step to restore mass conservation properties.

The core optimization objective in GMD is a regression loss that minimizes the distance between a generated sample and its displaced version, utilizing a stop-gradient operator to ensure the generator moves toward the prescribed transport step rather than altering the drift estimator itself.

Key Contributions and Results

1. Interpretation of the Score-Difference Drift (KL Flow)

The authors show that the simpler GMD instantiation (Algorithm 1) can be re-interpreted as a method for finding the fixed point of a WGF on the Kullback–Leibler (KL) divergence between Parzen-smoothed densities.

Correction of Prior Work: While Cao et al. (2026) claimed that the score-difference drift corresponds exactly to the WGF of the KL divergence between smoothed densities, this paper demonstrates that the velocity field expression used by Cao et al. is incorrect. The true WGF velocity for smoothed KL involves an integral of the log-ratio of densities, not a simple difference of smoothed scores. Consequently, the dissipation results claimed by Cao et al. are not established for the specific drift used in GMD.

2. Analysis of the Practical Algorithm (Sinkhorn Proxy)

The paper provides a detailed analysis of the algorithm actually used in practice (Deng et al., Algorithm 2), termed the Sinkhorn Proxy.

Fixed Point Property: The authors prove that the velocity field of the Sinkhorn Proxy is zero if and only if the source distribution $p$ and target distribution $q$ agree (for Gaussian kernels).
Non-Gradient Structure: A critical negative result is presented: the velocity field of the Sinkhorn Proxy does not generally correspond to a valid Wasserstein gradient flow of any distributional loss. For a flow to be a valid WGF, the velocity field must be a pure gradient. The authors prove that the Sinkhorn Proxy velocity field is a product of a spatially varying pre-conditioner and a score difference. This structure only satisfies the conditions of a WGF if the gradients of the pre-conditioner and the score difference are linearly dependent everywhere—a condition that does not hold in general.
Convergence Limitations: The paper demonstrates that the Sinkhorn Proxy fails to exhibit the desirable mass-transport dynamics expected of optimal transport metrics. In a synthetic setting with two distant, non-overlapping modes, the velocity field vanishes exponentially as the distance between modes increases relative to the kernel bandwidth. This indicates the method cannot effectively shift mass between distant modes, behaving more like a flow on the KL divergence (which suffers from mode collapse) rather than a true optimal transport distance.

3. Generalization to Other Flows

The authors extend the drifting framework to other energy functionals, showing that the general "drifting" methodology can be applied to:

Maximum Mean Discrepancy (MMD): Recovering the MMD drifting approach as the fixed point of the MMD WGF.
Sliced-Wasserstein (SW) Distance: Introducing a drifting formulation based on the SW flow.
f-Divergences (GAN Critics): Discussing drifting based on the Fenchel duals of f-divergences (e.g., KALE), offering alternatives to the KL-based approach.

Significance and Claims

The paper claims to provide the first rigorous WGF interpretation of the specific algorithm implemented by Deng et al. (2026), distinguishing it from the simpler theoretical variants discussed in prior literature.

Theoretical Clarification: It clarifies that while GMD shares structural similarities with WGFs on the Sinkhorn divergence, the practical algorithm is not a true gradient flow on that metric. This distinction is crucial for understanding the convergence properties and failure modes of the method.
Limitations Identified: The work highlights that the practical GMD algorithm inherits the "mode collapse" or "mass transport failure" issues common to score-based methods when modes are distant, rather than inheriting the robust mass-transport properties of true optimal transport distances.
Framework Extension: The paper establishes a general-purpose formulation (Algorithm 2 in the text) that allows drifting models to be constructed for any WGF, suggesting that the GMD framework is a versatile tool for generative modeling, provided the specific properties of the chosen divergence (e.g., MMD vs. KL) are understood.

The authors conclude that while the drifting paradigm offers a compelling alternative to adversarial training, its theoretical behavior is nuanced: the practical algorithm is a "proxy" that approximates a Sinkhorn flow but lacks the strict gradient structure and global convergence guarantees of a true optimal transport flow.