🤖 machine learning

SIPO: Stabilized and Improved Preference Optimization for Aligning Diffusion Models

The paper introduces SIPO, a stabilized and improved preference optimization framework for diffusion models that addresses training instability and off-policy bias through a novel gradient clipping mechanism and timestep-aware importance reweighting, demonstrating superior performance across image and video generation tasks.

Original authors: Xiaomeng Yang, Mengping Yang, Junyan Wang, Zhijian Zhou, Zhiyu Tan, Hao Li

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiaomeng Yang, Mengping Yang, Junyan Wang, Zhijian Zhou, Zhiyu Tan, Hao Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a talented artist (a Diffusion Model) how to paint pictures that humans actually like. The artist is already very good at creating images from scratch, but they don't always know exactly what you want. You want to give them feedback: "I like this picture more than that one."

This paper introduces a new teaching method called SIPO (Stabilized and Improved Preference Optimization). It fixes two major problems that happened when people tried to teach these artists using previous methods.

Here is the breakdown using simple analogies:

The Problem: The "Noisy Classroom"

The authors found that previous methods (like Diffusion-DPO) were like a chaotic classroom where the teacher shouted instructions at random times, confusing the student.

The "Bad Timing" Problem:
- The Analogy: Imagine the artist is painting a picture. They start with a blank canvas (very noisy) and slowly add details until the picture is clear.
- The Issue: The old method tried to give feedback on every single second of the painting process. But in the very beginning (the "early timesteps"), the canvas is just a blur of noise. Giving feedback here is like yelling "Draw a nose!" when the canvas is just a gray smudge. It's confusing and leads to the artist getting frustrated (training instability).
- The Fix: SIPO acts like a smart teacher who says, "Let's ignore the first 60 seconds when the canvas is just a blur. Let's only give feedback when the shapes start to appear." This stops the artist from getting confused by the noise.
The "Wrong Student" Problem:
- The Analogy: Imagine you are training a student using a textbook written by a different student. The textbook has the right answers, but the style is slightly different from how your student thinks.
- The Issue: The old method used "offline" data (pre-made examples) that didn't match exactly what the current artist was capable of doing. This created a gap between what the artist was learning and what they were actually doing, causing them to lose confidence or make weird mistakes later on (off-policy bias).
- The Fix: SIPO uses a "translator" (called importance re-weighting). It looks at every example and says, "This example is very similar to what our artist can do, so we'll listen closely. That other example is very different, so we'll lower its volume." This ensures the artist learns from the most relevant examples without getting overwhelmed by the mismatched ones.

The Solution: SIPO

The paper proposes SIPO as a two-step improvement:

DPO-C&M (Clipping and Masking): This is the "Smart Teacher." It masks (ignores) the early, noisy parts of the painting process where feedback isn't useful. It also clips (caps) the feedback so it doesn't get too extreme and scare the artist.
Timestep-Aware Reweighting: This is the "Translator." It adjusts the volume of the feedback based on how well the example matches the artist's current skill level. If an example is a perfect match, it speaks up. If it's a weird outlier, it whispers.

The Results: A Calmer, Better Artist

The authors tested this on both image generators (like Stable Diffusion) and video generators (like CogVideoX).

Stability: Previous methods were like a rollercoaster; the artist's performance would go up and down wildly, and sometimes crash completely after a few days of training. SIPO is like a smooth elevator; the artist improves steadily and doesn't crash.
Sensitivity: The old methods were very sensitive to a "temperature" setting (a knob called $\beta$ ). If you turned the knob slightly wrong, the artist would fail. SIPO is robust; it works well even if you don't tune the knob perfectly.
Quality: In head-to-head tests, SIPO produced images and videos that humans preferred more often than the old methods. It didn't just make "okay" pictures; it made pictures that were more consistent, colorful, and aligned with human taste.

Summary

In short, SIPO is a smarter way to train AI artists. Instead of shouting instructions at them while they are still confused by the noise, it waits for the right moment to speak and adjusts its voice to match the artist's current level. The result is a training process that is less likely to crash and produces better, more human-like art.

Technical Summary: SIPO (Stabilized and Improved Preference Optimization)

1. Problem Statement

While diffusion models have revolutionized image and video generation, aligning them with human preferences remains challenging. Existing alignment techniques, particularly Diffusion-DPO (Direct Preference Optimization), suffer from two fundamental issues that hinder stable and effective training:

Training Instability: The denoising process in diffusion models involves long-horizon trajectories with varied timesteps. The paper identifies that training instability arises primarily from early timesteps, which possess low importance weights and high gradient variances. This leads to non-monotonic loss dynamics, parameter sensitivity (specifically to the temperature hyperparameter $\beta$ ), and potential training collapse where performance degrades over time.
Off-Policy Bias: Preference optimization data is typically constructed from a fixed reference model or out-of-distribution sources rather than the current policy model. This creates a distributional gap where the training data captures a narrow subset of the state-action space, while the model must generalize across a broader space. This mismatch leads to inaccurate off-policy score estimation, reward hacking, and degraded generalization, particularly in video generation where temporal coherence is critical.

2. Methodology: SIPO Framework

The authors propose SIPO (Stabilized and Improved Preference Optimization), a framework designed to address the aforementioned instability and bias through a systematic analysis of diffusion trajectories and the introduction of importance sampling mechanisms.

2.1. Analysis of Diffusion Trajectories

Through systematic analysis on Stable Diffusion 1.5, the authors observed that:

Early timesteps (specifically $t \in [0, 63]$ ) correspond to low importance weights and introduce noisy gradients that destabilize training.
Mid-to-late timesteps provide the most informative signals for preference alignment.
Uniform timestep sampling underperforms compared to focusing on intermediate steps.

2.2. DPO-C&M (Clipping & Masking)

To address instability, the authors introduce DPO-C&M, which modifies the standard Diffusion-DPO objective:

Importance Weighting: An importance weight $w(t)$ is computed by comparing the learned reverse transition $p_\theta(x_{t-1}|x_t)$ with the forward posterior $q(x_{t-1}|x_t, x_0)$ .
Clipping and Masking: Timesteps with low importance weights (indicating unreliable or noisy gradients) are masked or clipped. Specifically, weights are clipped to a range $[1-\epsilon, 1+\epsilon]$ to prevent high variance. This acts as a soft mask to suppress gradients from regions where forward and reverse paths diverge significantly.

2.3. Timestep-Aware Importance Reweighting

To mitigate off-policy bias, SIPO incorporates a timestep-wise clipped importance reweighting scheme:

Off-Policy Correction: The framework reinterprets the preference optimization objective using importance sampling. It reweights updates based on the likelihood ratio between the current policy $p_\theta$ and the off-policy sampling distribution $q$ .
KL-Regularized Objective: The method formulates the optimization as minimizing the KL divergence between the current model and a "shaped" target distribution $p^*$ , which balances reward feedback with prior preferences.
Adaptive Step Sizes: The reweighted log-ratio term scales updates adaptively. It reinforces underestimated preferred samples and down-weights over-confident ones. This introduces a soft upper bound around the reference distribution, preventing reward hacking and reducing sensitivity to the $\beta$ hyperparameter.

The final SIPO loss function integrates these clipped importance weights into the pairwise preference signal at each denoising step, ensuring stable updates even when training on fixed, off-policy datasets.

3. Key Contributions

The paper outlines three primary contributions:

Systematic Analysis: A detailed investigation into diffusion-based preference optimization dynamics, revealing that early timesteps destabilize DPO training while mid-to-late timesteps drive performance gains.
SIPO Framework: The proposal of a principled, importance-weighted optimization framework that stabilizes training and mitigates off-policy mismatch by reweighting gradients according to sample likelihood and timestep quality.
Empirical Validation: Extensive experiments demonstrating that SIPO consistently stabilizes the preference optimization process and outperforms existing alternatives (e.g., Diffusion-DPO, DPO-C&M) across both image and video generation tasks.

4. Experimental Results

The authors evaluated SIPO on various baseline models, including Stable Diffusion 1.5, SDXL, CogVideoX-2B/5B, and Wan2.1-1.3B.

Text-to-Video: On CogVideoX-2B, Diffusion-DPO showed severe degradation after 1000 steps (Total VBench score dropping from 80.91 to 67.28), whereas SIPO remained stable and improved (81.68). SIPO consistently outperformed Diffusion-DPO and DPO-C&M across Total, Quality, and Semantic scores on all video models tested.
Text-to-Image: On SD1.5 and SDXL using the Pick-a-Pic dataset, SIPO achieved the highest scores on three out of four metrics (PickScore, HPSv2, Aesthetics) compared to Diffusion-DPO and other baselines like Diffusion-KTO and SPO.
Parameter Sensitivity: SIPO demonstrated significantly higher robustness to the temperature parameter $\beta$ . While Diffusion-DPO performance fluctuated wildly with different $\beta$ values, SIPO maintained consistent performance across a wide range.
Human Evaluation: In a study with 200 prompts and 12 annotators, SIPO achieved a 67% win rate (first-place rankings), significantly outperforming Diffusion-DPO and the base models.
Online/Iterative Learning: In iterative training scenarios, SIPO maintained stable performance, whereas Diffusion-DPO exhibited degradation due to reward hacking and distributional drift.

5. Significance and Claims

The paper claims that SIPO provides valuable guidelines for improved preference optimization by emphasizing the importance of timestep-aware alignment. The authors assert that their method:

Stabilizes Training: By identifying and masking uninformative early timesteps, SIPO prevents the training collapse often seen in Diffusion-DPO.
Mitigates Off-Policy Bias: The importance reweighting scheme allows for effective training on fixed, offline datasets without the need for complex on-policy exploration.
Generalizes Well: The approach is effective across different model architectures (including flow-based models like WanX) and modalities (image and video).

The authors conclude that SIPO offers a scalable and principled approach to aligning diffusion-based generation with human preferences, making outputs more consistent and controllable without requiring new generative capabilities beyond the underlying models. They recommend pairing this optimization with careful dataset curation and safety filtering.