📊 statistics

Variance-Tilted Diffusion Models for Diverse Sampling

This paper introduces a variance-tilted diffusion sampling method that utilizes a Doob $h$ -transform to generate diverse candidate sets by explicitly repelling posterior denoised means and guiding particles toward regions of higher feature variance.

Original authors: Iskander Azangulov, Leo Zhang, Kianoosh Ashouritaklimi

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Iskander Azangulov, Leo Zhang, Kianoosh Ashouritaklimi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magical artist (a "Diffusion Model") who is incredibly good at painting pictures based on a description, like "a duck made of glass." Usually, if you ask this artist to paint 100 ducks, they will just paint 100 slightly different versions of the same perfect duck. They all look very similar, standing in the same pose, with the same lighting.

This is great if you just want one good picture. But what if you need a whole collection of ducks for a game, and you need them to be totally different from each other? Maybe one is swimming, one is flying, one is made of green glass, and another is made of blue glass?

The standard way of asking the artist to do this usually fails. It's like asking a chef to make 100 different cakes, but they just make 100 identical chocolate cakes because that's their favorite recipe.

The Problem: The "Groupthink" Artist

The paper argues that current AI models are too "groupthink-y." They are trained to find the most probable, safe answer. When you ask for a batch of samples, they just repeat the same safe answer over and over.

The Solution: The "Variance-Tilted" Method

The authors propose a new way to talk to the artist. Instead of just saying, "Paint a duck," they add a special rule: "Paint 100 ducks, but make sure that when you look at them all together, they are spread out as far as possible."

They call this "Variance-Tilted Diffusion."

Here is how it works, using a simple analogy:

1. The Feature Map (The "Measuring Tape")

First, you need a way to measure "difference." The authors use a tool called a Linear Operator (A). Think of this as a special measuring tape.

It could measure the shape of the duck.
It could measure the color.
It could measure the background.
It could even measure just the left wing.

This tool translates the complex image of a duck into a simple list of numbers (features) that represent what makes that duck unique.

2. The Variance Goal (The "Spread Out" Rule)

The goal is to maximize the Variance. In everyday terms, variance is just a fancy word for "spread."

If all 100 ducks are standing in a straight line, the variance is low (they are clumped together).
If the ducks are scattered across the whole room, the variance is high.

The paper creates a mathematical rule that says: "We want the final batch of ducks to have the highest possible spread according to our measuring tape."

3. The Doob H-Transform (The "Ghostly Guide")

This is the tricky math part, but here is the simple version.
Usually, the AI generates images one by one, independently. To make them diverse, the authors change the rules of the game. They imagine that the AI isn't just painting one duck; it's painting a whole group simultaneously, and the group is "talking" to each other.

They use a mathematical trick called a Doob h-transform. Think of this as giving the artist a Ghostly Guide that whispers in their ear while they paint.

The Whisper: "Hey, that duck you're painting looks a lot like the one you painted 5 seconds ago. Move it a bit! Make it stand on one leg instead of two!"
The Repulsion: The guide pushes the new ducks away from the others (like magnets with the same pole).
The Curvature: The guide also nudges them toward areas where the "spread" is naturally higher, ensuring they don't just cluster in one weird corner.

The Two Forces at Play

The paper breaks this "Ghostly Guide" into two simple forces:

The "Don't Copy Me" Force: This pushes the ducks apart. If the AI tries to paint a duck that looks too much like the average of the group, this force pushes it away. It ensures the ducks don't all look like the same "average" duck.
The "Explore the Edge" Force: This pushes the ducks toward the edges of the possible world. It encourages the AI to try things that are slightly unusual or rare, rather than sticking to the boring, safe center.

The Result

When the authors tested this on images (like "a glass duck" or "a corgi with a ball"), the results were striking:

Standard AI (CFG): Produced a group of ducks that all looked very similar.
Their Method (Variance-Tilted): Produced a group of ducks with wildly different poses, backgrounds, and styles. Some were swimming, some were flying, some were in the rain, some in the sun.

Why This Matters (According to the Paper)

The paper claims this is better than previous methods because:

It's Exact: They didn't just guess a rule to make things diverse. They started with the goal (maximize spread) and mathematically derived the exact steps to get there.
It's Transparent: You can see exactly what the "Ghostly Guide" is doing. It's not a black box; it's a clear push-and-pull between the individual ducks and the group.

In short: The paper teaches the AI to stop being a copycat and start being a creative director, ensuring that when you ask for a batch of ideas, you get a rich, diverse collection instead of 100 copies of the same thing.

Technical Summary: Variance-Tilted Diffusion Models for Diverse Sampling

Problem Statement
Diffusion models have become the standard for generative modeling across domains ranging from high-resolution image synthesis to scientific discovery (e.g., protein folding and molecular docking). However, the standard sampling paradigm relies on drawing independent and identically distributed (i.i.d.) samples. This approach is often suboptimal for downstream tasks requiring a diverse set of candidates that cover distinct regions of the data manifold, such as generating structurally diverse ligands in drug design or robust trajectory proposals in robotic motion planning.

Existing methods to enforce diversity, such as particle guidance, introduce repulsive potentials into sampling dynamics. While effective, these methods modify the drift heuristically, resulting in an implicit target distribution that is computationally intractable to evaluate or control precisely.

Methodology
The authors propose a "target-first" formulation where the batch target distribution is explicitly defined before deriving the exact sampler. The core innovation is a variance-weighted batch distribution that favors collections of samples with large empirical spread after a prescribed linear feature map.

For a batch of $n$ samples $\mathbf{x} = (x^{(1)}, \dots, x^{(n)})$ , the target distribution $\pi^A_0(\mathbf{x})$ is defined as:
$\pi^A_0(\mathbf{x}) \propto \text{Var}^A_n(\mathbf{x}) \prod_{i=1}^n p_0(x^{(i)})$
where $p_0$ is the original data distribution, $A: \mathbb{R}^d \to \mathbb{R}^k$ is a linear operator mapping samples to a task-specific feature space, and $\text{Var}^A_n(\mathbf{x})$ is the empirical variance of the features $A(x^{(i)})$ .

To sample from this distribution, the authors derive the exact Doob $h$ -transform of the independent diffusion dynamics. This involves computing the conditional expectation of the batch variance given the noisy state at time $t$ :
$h^A_t(\mathbf{x}_t) := \mathbb{E}[\text{Var}^A_n(\mathbf{X}_0) \mid \mathbf{X}_t = \mathbf{x}_t]$

Using Tweedie's identities and a bias-variance decomposition, the authors derive a closed-form expression for the score correction $\nabla_{\mathbf{x}_t} \log h^A_t(\mathbf{x}_t)$ . This correction consists of two distinct terms:

Interaction Term (Repulsion): A term that repels the posterior denoised means ( $\mu_t$ ) of the particles from each other in the feature space. This is computed efficiently using vector-Jacobian products (VJP) through the denoiser.
Curvature Term: A term that moves particles toward regions of higher feature variance, involving the divergence of the score function (specifically $\text{div}(A^\top A \nabla s_t)$ ). This is estimated via finite differences or stochastic trace estimators (e.g., Hutchinson estimators).

The resulting sampler is an interacting-particle process where the score for each particle is the sum of the standard score and the derived Doob correction.

Key Contributions

Target-First Formulation: The paper frames diverse generation as an exact sampling problem from an explicit batch target distribution, avoiding the uncontrolled repulsive forces of heuristic methods.
Closed-Form Doob Correction: The authors provide an analytical derivation of the score correction for a general linear operator $A$ , decomposing it into a repulsion term for posterior means and a curvature term for feature variance.
Tractable Sampling Algorithm: A discretized interacting-particle sampler is introduced. The leading interaction term leverages efficient VJP computations, while the curvature term is handled via numerical estimation, making the method practical for existing diffusion models.

Results
The authors evaluated the approach on image generation using Stable Diffusion v1.5 (512x512) with various prompts. They compared three configurations:

Standard Classifier-Free Guidance (CFG).
Variance-Tilted (VT) without the divergence term.
Variance-Tilted + Divergence (VT + Divergence).

Qualitative results (Figures 1–5) demonstrate that the proposed methods produce a more diverse set of samples compared to standard CFG. The generated images cover a wider range of poses, backgrounds, and visual styles rather than merely producing minor variations of high-probability modes. The inclusion of the divergence term further refines the diversity, though the authors note that the full method can slightly negatively impact image quality and increases computational cost due to divergence estimation.

Significance and Claims
The paper claims to offer a probabilistic alternative to heuristic particle repulsion. By deriving the exact sampler for a variance-weighted target, the method provides a transparent probabilistic objective rather than an implicit one. The authors position this as a rigorous approach to diverse sampling that retains practical implementation paths.

The authors remain modest regarding limitations, acknowledging that while the method qualitatively improves diversity, it may slightly degrade image quality and incurs higher computational costs due to the estimation of second-order derivatives (divergence). They suggest future work could focus on annealing the Doob correction to mitigate quality issues and developing more efficient estimators for the curvature term.