📊 statistics

Towards More General Control of Diffusion Models Using Jeffrey Guidance

This paper introduces Jeffrey guidance, a principled framework that leverages Jeffrey's rule to extend diffusion model control beyond standard conditional sampling by updating marginal distributions toward prescribed targets while minimally perturbing the joint distribution, thereby achieving improved image quality and fairness.

Original authors: Raphaël Razafindralambo, Rémy Sun, Frédéric Precioso, Jes Frellsen, Pierre-Alexandre Mattei

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Raphaël Razafindralambo, Rémy Sun, Frédéric Precioso, Jes Frellsen, Pierre-Alexandre Mattei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very talented artist (a Diffusion Model) who can paint beautiful pictures from scratch. This artist is great, but sometimes you want to tweak their work without hiring a new artist or retraining them from scratch. You just want to say, "Hey, make the sky bluer," or "Make sure there are equal numbers of men and women in the crowd."

Usually, artists have a simple way to do this: you give them a specific label, like "Cat," and they try to paint a cat. This is called Classifier Guidance. But what if you don't want a specific label? What if you want to change the overall vibe of the painting, like making the colors match a specific photo album, or ensuring the crowd isn't biased toward one gender? Standard instructions struggle with these vague, "big picture" goals.

This paper introduces a new method called Jeffrey Guidance. Think of it as a "Universal Remote Control" for the artist's imagination.

The Core Idea: The "Jeffrey Rule"

To understand the magic, imagine you have a recipe book (the Joint Distribution) that tells you how to make a dish (the image) based on the ingredients you have (the attributes, like gender or age).

Old Way (Bayes/Standard Guidance): If you want a specific dish, you say, "I want the recipe for Spaghetti." The artist looks up "Spaghetti" and cooks that. It's rigid.
New Way (Jeffrey Guidance): Imagine you don't care about the specific recipe, but you want the flavor profile of the final meal to match a specific target (like "I want the crowd to be 50% men and 50% women"). Jeffrey's rule is a mathematical trick that says: "Keep the cooking method exactly the same, but swap out the ingredients to match the new flavor profile you want."

It updates the "marginal" (the overall mix of ingredients) to match your target, while keeping the "conditional" (how the artist actually paints the details) intact. It's the most minimal change possible to get the result you want.

Analogy 1: The DJ and the Playlist (Embedding Guidance)

Imagine the artist is a DJ. The "embedding" is like the genre of music they are playing.

The Problem: The DJ usually plays a mix that sounds a bit different from the original hits (the training data). The "FID" score is like a music critic rating how close the DJ's mix is to the original hits.
The Jeffrey Fix: The authors used Jeffrey Guidance to tell the DJ, "Don't just play any song; make the entire playlist sound exactly like the original Top 40 hits."
The Result: The DJ didn't change their style of mixing (the neural network wasn't retrained). They just adjusted the volume of certain tracks during the performance. The result? The playlist sounded much more like the original hits (the FID score dropped significantly), even though the individual songs still sounded like the DJ's work.

Analogy 2: The Seating Chart (Fairness and Decorrelation)

Imagine the artist is hosting a party and generating a guest list.

The Problem: In the original guest list, the artist accidentally invited way more "Young Women" than anyone else, and almost no "Old Men." Also, the "Young" and "Male" guests were strangely linked (maybe the artist only thought of young men as "athletes" and young women as "dancers").
The Goal: The organizers want a fair party. They want:
1. Gender Parity: Exactly 50% men and 50% women.
2. Decorrelation: Being "Young" shouldn't automatically mean you are "Male" or "Female." They should be independent.
The Jeffrey Fix: Instead of telling the artist, "Draw a man," or "Draw a woman," the authors used Jeffrey Guidance to say, "Adjust the guest list so the overall ratio of men to women is 50/50, and break the link between age and gender."
The Result:
- Parity: The party became perfectly balanced (50/50) without ruining the quality of the invitations (the images still looked good).
- Decorrelation: The strange link between "Young" and "Male" was broken. The artist started inviting young men, young women, old men, and old women in a way that felt natural and independent, rather than following a hidden bias.

Why is this a big deal?

It's Plug-and-Play: You don't need to retrain the artist. You just add a small "correction term" while the artist is working.
It's Flexible: You aren't limited to simple labels like "Cat" or "Dog." You can aim for complex goals like "Match the statistical distribution of a specific dataset" or "Remove bias between two specific traits."
It's Principled: It's not just a random guess or a hack. It's based on a solid mathematical rule (Jeffrey's Rule) that guarantees you are making the smallest possible change to the original model to achieve your new goal.

The Catch

The paper notes that while this method works wonders for the "stats" (the numbers look perfect), sometimes the human eye doesn't notice a huge difference in the pictures themselves. It's like tuning a radio: the signal (the statistics) is perfect, but the song (the image) might sound mostly the same to a casual listener. However, for tasks like fairness or matching specific data distributions, this "invisible tuning" is exactly what's needed.

In short, Jeffrey Guidance gives us a precise, mathematical way to steer AI art generators toward specific statistical goals—like fairness or matching a dataset—without having to rebuild the engine from scratch.

Technical Summary: Towards More General Control of Diffusion Models Using Jeffrey Guidance

1. Problem Statement

Diffusion models offer significant flexibility in generative tasks, allowing output control at sampling time through guidance mechanisms without retraining the underlying neural network. While standard guidance techniques (e.g., classifier guidance) effectively handle conditional sampling for specific classes, they often fail to address more complex objectives where the target distribution is implicit or defined only through heuristics.

Current approaches for tasks such as fairness, concept control, and compositional generation frequently rely on heuristic modifications to sampling dynamics. These methods tilt the original distribution but lack an explicitly defined target distribution, making the actual objective difficult to characterize or interpret. There is a need for a principled framework that can update marginal distributions toward a prescribed target while preserving the conditional structure of the model, thereby enabling control over complex statistical properties (e.g., attribute independence or embedding distribution matching) that standard conditioning cannot express.

2. Methodology: Jeffrey Guidance

The paper proposes Jeffrey guidance, a framework that leverages Jeffrey's rule of conditioning to update diffusion models. Unlike Bayes-based approaches that condition on specific events (standard conditioning), Jeffrey's rule allows for conditioning on complex predicates, such as matching a specific marginal distribution.

Theoretical Foundation

Given a joint model $p(x, y)$ where $x$ represents the image space and $y$ represents attributes/features, the goal is to update the marginal $p(y)$ to a target distribution $p^\star(y)$ while preserving the conditional structure $p(x|y)$ . Under Jeffrey's rule, the updated joint distribution is defined as:
$p^\star(x, y) = p(x|y)p^\star(y)$
This update minimizes the perturbation to the original joint distribution (an information projection) while satisfying the marginal constraint.

The paper derives a reweighting formulation for the marginal over $x$ :
$p^\star(x) = p(x) \int p(y|x)\rho(y) dy$
where $\rho(y) = p^\star(y)/p(y)$ is the density ratio. In cases where features are deterministically derived from images ( $y = u(x)$ ), this simplifies to:
$p^\star(x) = p(x)\rho(u(x))$

Implementation in Diffusion Models

To apply this to score-based diffusion models, the authors derive a guided score function. The standard score $\nabla_{x_t} \log p_t(x_t)$ is augmented with a correction term based on the density ratio:
$\nabla_{x_t} \log \tilde{p}^\star_t(x_t) = \nabla_{x_t} \log p_t(x_t) + \nabla_{x_t} \log \int p(y|\hat{x}_0(x_t, t))\rho(y) dy$
where $\hat{x}_0(x_t, t)$ is an estimate of the clean sample obtained via Tweedie's formula.

For deterministic feature mappings (e.g., embedding spaces), the guidance simplifies to:
$\nabla_{x_t} \log \tilde{p}^\star_t(x_t) = \nabla_{x_t} \log p_t(x_t) + \nabla_{x_t} \log \rho(u(\hat{x}_0(x_t, t)))$

The framework introduces a hyperparameter $\lambda$ to scale the correction term, controlling the trade-off between sample quality and faithfulness to the target distribution. The density ratio $\rho$ is estimated using a binary classifier trained to distinguish between samples from the target distribution and the model's current distribution.

3. Key Contributions

The paper makes three primary contributions:

Jeffrey Guidance Framework: A novel, plug-and-play guidance method that generalizes standard classifier guidance. It uses Jeffrey's rule to match generated output distributions to arbitrary target distributions, enabling control over objectives previously inaccessible to standard guidance.
Embedding Distribution Matching: A proof of concept demonstrating the ability to match model output distributions to a target distribution in an embedding space (specifically Inception embeddings). This application significantly reduces the Fréchet Inception Distance (FID) without requiring retraining.
Attribute Distribution Control for Fairness: A demonstration of controlling image attribute distributions to achieve fairness objectives. The method successfully enforces gender parity (distribution matching) and decorrelates attributes (e.g., "Male" and "Young") in generated faces, tasks that standard classifier guidance cannot address directly.

4. Experimental Results

The authors evaluated Jeffrey guidance on CIFAR-10, FFHQ, and CelebA-HQ datasets.

Embedding Guidance (FID Reduction)

Setup: The method targeted the Inception embedding distribution of the training data.
Results: On CIFAR-10, FID improved from $\approx 5.88$ to $\approx 2.55$ (training set) and from $\approx 9.76$ to $\approx 6.71$ (test set). On FFHQ, FID dropped from $\approx 20.13$ to $\approx 12.91$ .
Observation: The authors note that while FID decreased drastically, visual changes to the generated images were minimal, suggesting limitations in FID as a purely perceptual metric. The best results were achieved by applying guidance only to the final denoising steps ( $\delta=10$ ).

Fairness Applications

Gender Parity (Distribution Matching): On CelebA-HQ, the method adjusted the proportion of "Male" attributes from $\approx 0.296$ to $\approx 0.522$ (targeting 0.5), reducing the Fairness Discrepancy (FD) from $0.288$ to $0.032$. This was achieved with minimal degradation in FID ($21.61$ vs. baseline $22.30$).
Attribute Decorrelation: The method successfully decorrelated "Male" and "Young" attributes, reducing the Pearson correlation coefficient from $-0.541$ to near zero ($0.005$) at $\lambda=3.0$ . This was achieved by reducing the dominant joint mode (young female samples) while maintaining marginal distributions relatively stable, with a minor FID increase of approximately one point.
Comparison: Compared to a baseline of ancestral sampling combined with standard guidance, Jeffrey guidance achieved similar fairness metrics with lower FID costs and modified fewer samples, as it balances class directions continuously along the sampling trajectory rather than pushing samples entirely toward a single class.

5. Significance and Claims

The paper positions Jeffrey guidance as a principled extension of diffusion model control. Its significance lies in:

Generalization: It recontextualizes standard classifier guidance as a special case of Jeffrey conditioning, thereby unifying these approaches under a broader theoretical framework.
Expressiveness: It enables the specification of explicit target distributions for complex objectives (like attribute independence) that were previously only achievable through heuristic or implicit methods.
Practicality: The method requires no retraining of the diffusion model, only the addition of a correction term during sampling and the training of a lightweight density ratio estimator.

The authors maintain a modest tone, framing the work as a "proof of concept." They acknowledge that the current implementation relies on a classifier to estimate density ratios, similar to how early classifier guidance worked, and express hope that future work will evolve toward classifier-free variants. They also note that while the method offers new expressive power for beneficial applications (fairness, memorization mitigation), it also carries the risk of empowering malicious actors to generate nefarious outputs.