🤖 machine learning

Linear-DPO: Linear Direct Preference Optimization for Diffusion and Flow-Matching Generative Models

This paper proposes Linear-DPO, a unified preference optimization framework that derives a generalized objective for both diffusion and flow-matching models by replacing the standard sigmoid-based utility with a linear utility and an EMA-updated reference model to overcome objective mismatches and improve text-to-image generation alignment.

Original authors: Kesong Li, Yixuan Xu, Kuo-kun Tseng, Weiyi Lu, Kan Liu, Tao Lan

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kesong Li, Yixuan Xu, Kuo-kun Tseng, Weiyi Lu, Kan Liu, Tao Lan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very talented artist who can paint pictures based on your descriptions. This artist has learned by looking at billions of images and text captions from the internet. They are great at making things look real, but they don't always know what humans actually prefer. Sometimes they make pictures that are technically correct but a bit weird, ugly, or just not what you asked for.

To fix this, we need to teach the artist to listen to human feedback. This paper introduces a new, smarter way to do that teaching, called Linear-DPO.

Here is a simple breakdown of how it works, using some everyday analogies:

1. The Problem: The "One-Size-Fits-All" Teacher

Previously, researchers tried to teach these AI artists using a method called DPO (Direct Preference Optimization). Think of DPO like a strict teacher who says, "If you get the answer right, great! If you get it wrong, stop trying immediately."

The Issue: This "stop immediately" rule works well for language models (like chatbots) where you just need to pick the right word. But for image generation, it's like trying to sculpt a statue by only chipping away the big chunks and then quitting the moment the shape looks okay. You never get the fine details right because the teacher stops giving feedback too early.
The Gap: Also, the old method only worked for one specific type of AI artist (called "Diffusion" models). Newer, faster artists (called "Flow-Matching" models) were left out of the class entirely.

2. The Solution: A Unified Classroom

The authors of this paper realized that both the old "Diffusion" artists and the new "Flow-Matching" artists are actually doing the same thing, just in slightly different ways. They built a Unified Framework.

The Analogy: Imagine the old method was a teacher who only spoke to students wearing blue shirts. The new method is a teacher who speaks a universal language that both the "blue shirt" students and the "red shirt" students understand perfectly. This allows them to train the newest, most powerful AI artists (like SD3) for the first time using preference learning.

3. The Secret Sauce: The "Linear" Utility

The biggest innovation is changing how the teacher gives feedback.

The Old Way (Sigmoid): The old method used a "Sigmoid" function. Imagine a light switch. It's either OFF (0) or ON (1). Once the student gets the answer "good enough," the switch flips to ON, and the teacher stops correcting them. This causes the student to get stuck at a "good enough" level and never improve further.
The New Way (Linear-DPO): The authors replaced the light switch with a dimmer switch (a linear function).
- How it works: Even when the student is doing well, the teacher keeps giving small, gentle nudges. It's like a coach who says, "Good job, but let's make the colors a tiny bit brighter," even after the painting is already finished.
- The Result: This keeps the learning process smooth and continuous. The artist doesn't stop improving just because they hit a "good" score; they keep refining the details until the picture is truly beautiful.

4. The "Moving Target" Reference

In training, you usually compare the student's work to a "reference" (a baseline model) to make sure they don't go off the rails.

The Old Way: The reference was a frozen, static statue. Once the student got better than the statue, the comparison became useless.
The New Way: The authors use an EMA (Exponential Moving Average) reference. Imagine the reference is a shadow that slowly follows the student. As the student gets better, the shadow moves with them. This ensures the student is always being challenged to do slightly better than their current self, preventing them from getting lazy or stuck.

5. The Results

The paper tested this new method on several famous AI artists (SD1.5, SDXL, and the newer SD3).

The Outcome: The artists trained with Linear-DPO produced images that humans rated as significantly better. They looked more realistic, followed instructions more closely, and had richer details.
The Proof: In head-to-head tests, the new method beat the old methods (like standard DPO or simple fine-tuning) almost every time, especially on the newest, most powerful models.

Summary

Think of Linear-DPO as upgrading from a teacher who uses a light switch (on/off, stops too early) to a teacher with a dimmer switch (smooth, continuous feedback). They also made sure this teacher can teach all types of AI artists, not just the old ones. The result is AI-generated art that looks much more like what humans actually want to see.

Technical Summary: Linear-DPO

Problem Statement

Direct Preference Optimization (DPO) has proven effective for aligning Large Language Models (LLMs) with human preferences but faces significant challenges when applied to text-to-image (T2I) generation. Existing approaches, such as Diffusion-DPO, are limited by two primary issues:

Model Architecture Limitation: Current theoretical analyses and implementations are confined to denoising diffusion models (e.g., SD1.5, SDXL) based on U-Net architectures. They overlook flow-matching models (e.g., SD3-Medium, Qwen-Image), which utilize Rectified Flow and MMDiT architectures and have become the state-of-the-art for high-resolution synthesis.
Objective Mismatch: The standard DPO objective, derived from discrete language modeling, relies on a sigmoid-based utility function that encourages "margin maximization." In this paradigm, gradients vanish rapidly once the probability gap between chosen and rejected samples is sufficiently large. This behavior is ill-suited for regression-based generative tasks (diffusion and flow-matching), which require long-horizon, stable, small-step updates to continuously refine fine-grained visual details. The rapid gradient decay in standard DPO leads to a "pseudo-convergence" trap, causing models to become rigid and cease refining details early in training.

Methodology

1. Unified SDE Framework for Diffusion and Flow-Matching

The authors derive a generalized DPO objective that unifies diffusion and flow-matching models under a single Stochastic Differential Equation (SDE) framework.

Reinterpreting Flow-Matching: While flow-matching is typically defined by a deterministic Ordinary Differential Equation (ODE), the authors reformulate it as a velocity-based SDE. By introducing necessary stochasticity, they derive conditional probabilities ( $q(x_{t-1}|x_t)$ ) for flow-matching that are mathematically equivalent to those in diffusion models.
Unified Loss Formulation: This allows the derivation of a single DPO loss function (Eq. 11) applicable to both paradigms. The loss is a logistic function of the difference between the policy model's squared error and the reference model's squared error on the prediction target (noise $\epsilon$ for diffusion, velocity $v$ for flow-matching).

2. Gradient Analysis and Linear-DPO

Through a gradient perspective analysis, the authors identify that standard DPO acts as a weighted Supervised Fine-Tuning (SFT) where the weights are modulated by a sigmoid function. They argue this is suboptimal for generative tasks due to:

Overly Large Updates: The scaling factor $\bar{\beta}$ in diffusion contexts is much larger than in NLP, causing volatile gradients.
Coupled Hyperparameters: The same parameter controls both the discrimination temperature and the update magnitude.
Premature Saturation: The sigmoid function saturates quickly, halting learning before fine-grained refinement is complete.

To address this, Linear-DPO is proposed with two core modifications:

Linear Utility Function: The aggressive sigmoid gating is replaced with a sustained linear utility function ( $u_{linear}(x) = 0.2x + 0.5$ ), clipped to a range $[\eta, 1]$ . This ensures that even when preferences are separated, the model continues to receive small, persistent gradients to refine details, avoiding the "pseudo-convergence" trap.
EMA-Updated Reference Model: Instead of using a fixed reference model (which can hinder improvement once the policy outperforms it) or abrupt updates, Linear-DPO maintains an Exponential Moving Average (EMA) of the policy model as the reference. This encourages continuous optimization and stabilizes training.

Key Contributions

Generalized DPO Objective: The establishment of a unified DPO formulation that bridges diffusion and flow-matching generative models via a reverse-time SDE perspective.
Gradient Analysis and Linear-DPO: A theoretical analysis revealing the suboptimality of sigmoid-based objectives for regression-based generative tasks, leading to the proposal of Linear-DPO with a linear utility function and EMA reference updates.
Empirical Validation: Demonstration of Linear-DPO's effectiveness and scalability across a spectrum of models, from standard diffusion (SD1.5, SDXL) to advanced flow-matching models (SD3-Medium).

Experimental Results

The authors evaluated Linear-DPO on SD1.5, SDXL, and SD3-Medium using datasets like Pick-a-Pic v2 and a high-quality subset of HPDv3.

Quantitative Performance: Linear-DPO consistently outperformed baselines (SFT, Diffusion-DPO, Diffusion-KTO, DSPO, MaPO, DMPO) across multiple automated metrics, including PickScore, HPSv2/v3, Aesthetics, CLIP, and Image Reward. Notably, it achieved state-of-the-art results in human preference alignment and visual quality.
Qualitative Improvements: Visual comparisons showed that Linear-DPO significantly reduced structural warping and improved text-image alignment in SD1.5. For the high-capacity SD3-M, it further enhanced aesthetic quality, color harmony, and background details.
Ablation Studies: Experiments confirmed that the linear utility function outperformed other utility forms (Kahneman-Tversky, Loss-Averse, Risk-Seeking). Additionally, the use of an EMA reference model with a decay factor of $\gamma=0.995$ yielded the best performance, preventing the instability associated with fixed or abruptly updated references.

Significance

The paper claims that Linear-DPO offers a principled path for preference learning in large-scale generative models. By addressing the theoretical gap between diffusion and flow-matching and correcting the optimization mismatch in standard DPO, the method provides a versatile framework that supports both superior alignment performance and scalability. The authors position Linear-DPO as a solution to the "pseudo-convergence" problem, enabling models to sustain refinement over long training horizons, which is critical for high-fidelity image generation.