🤖 AI

MixFlow Training: Alleviating Exposure Bias with Slowed Interpolation Mixture

This paper introduces MixFlow, a novel training approach that alleviates exposure bias in diffusion models by leveraging a "slowed interpolation mixture" strategy to better align training and testing conditions, thereby achieving state-of-the-art image generation results on ImageNet.

Original authors: Hui Li, Fu-Yun Wang, Haoyuan Xia, Jiayue Lyu, Kaihui Cheng, Siyu Zhu, Jingdong Wang

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hui Li, Fu-Yun Wang, Haoyuan Xia, Jiayue Lyu, Kaihui Cheng, Siyu Zhu, Jingdong Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to draw a picture by slowly turning a knob that changes a cloud of static noise into a clear image. This is how modern "diffusion" models work. They learn by looking at a photo, adding static noise to it, and then trying to guess how to remove that noise step-by-step to get the original picture back.

Here is the problem: During training (the learning phase), the robot is a bit of a cheat. Every time it tries to guess the next step, it is shown the perfect version of the noisy picture from the previous step. It's like a student taking a test where the teacher whispers the correct answer for the previous question before they even write down their own.

But during testing (when the robot actually draws a new picture), the teacher stops whispering. The robot has to use its own previous guess to figure out the next step. If it makes a tiny mistake early on, that mistake gets carried forward, gets bigger, and eventually, the picture looks weird or blurry. This is called "exposure bias"—the robot is exposed to perfect data while learning, but messy data when it's on its own.

The "Slow Flow" Discovery

The authors of this paper noticed something strange happening when they watched the robot draw. They found that when the robot generated a noisy picture at a certain moment (let's call it "Time T"), the perfect version of that noise (the one the robot should have been looking at) actually belonged to an earlier time in the process, a time with more noise.

Think of it like this: Imagine you are running a race. At the 10-second mark, you look back and realize the "perfect" version of your position actually happened at the 8-second mark. You are effectively "slower" than the ground truth. The robot's generated path is lagging behind the perfect path, stuck in a "slowed" state with more noise than it should have.

The MixFlow Solution

Instead of trying to fix the robot's running style after the race (which other methods tried), the authors decided to change how the robot learns the race. They introduced a new training method called MixFlow.

In standard training, the robot only looks at the perfect noise from the exact moment it's supposed to be at. In MixFlow, the robot is trained to look at a "mixture" of perfect noises. It looks at the perfect noise for the current moment, but also the perfect noises from those "slowed" moments with more noise.

The Analogy:
Imagine you are learning to ride a bike.

Standard Training: You only practice on a perfectly flat, smooth road. When you go out in the real world with bumps and wind, you crash because you weren't prepared for the lag.
MixFlow Training: You practice on the smooth road, but you also practice on a slightly bumpier, "slower" version of the road that mimics the lag you'll feel later. You learn to handle the "slowed" feeling while you are still in the safety of training.

By training on this "slowed interpolation mixture," the robot learns to handle the fact that its own guesses will be slightly "slower" and noisier than the perfect ground truth. It stops getting confused when it has to rely on its own previous guesses.

What They Ruled Out

The authors were very careful to test what doesn't work. They tried a method called "Input Perturbation," which involves adding random noise to the training data to simulate mistakes. They found that if you just add random noise without understanding the "slow flow" pattern, you might accidentally teach the robot with "faster" (less noisy) data that doesn't match reality, or "slower" data that is too far off. Their experiments showed that blindly adding noise often makes things worse, whereas their specific "slowed" mixture is the key.

They also compared their method to fixing the problem during the drawing process (inference) by tweaking the steps. They found that while those tweaks help a little, they are like trying to steer a car that's already drifting; it's better to fix the engine (the training) in the first place.

The Results: How Sure Are We?

The authors didn't just guess; they ran massive experiments on real image datasets.

The Proof: They tested their MixFlow method on several top-tier image generators (like SiT, REPA, and RAE) using the ImageNet dataset, which contains 1.2 million images of 1,000 different object categories.
The Numbers: On the 256 × 256 pixel images, their MixFlow model achieved a score of 1.43 (without guidance) and 1.10 (with guidance). On the larger 512 × 512 images, it hit 1.55 (without guidance) and 1.10 (with guidance).
- Note: In this field, a lower "FID" score is better. These numbers are incredibly low, meaning the generated images are almost indistinguishable from real photos.
Text-to-Image: They also tested it on a text-to-image model (SD 3.5). When asked to draw complex scenes, like "five savory pizzas" or "a bird on the left of a clock," the MixFlow version followed the instructions much better than the standard version, getting the counts and positions right.

The paper suggests that this approach is a significant step forward because it directly addresses the root cause of the error accumulation. It's not a magic trick; it's a smarter way of practicing. By letting the robot practice on "slowed" versions of the truth, it becomes much more robust when it has to generate images on its own.

In short, the paper shows that by changing the training diet to include "slowed" data, diffusion models can stop stumbling over their own feet and start running a much smoother race.

Technical Summary: MixFlow Training

Problem Statement

This paper addresses the training-testing discrepancy, commonly known as exposure bias, in diffusion and flow matching models. During training, prediction networks receive ground-truth noisy data (an interpolation of noise and data) as input at each timestep. Conversely, during testing (sampling), the network receives generated noisy data from the previous step. This discrepancy leads to prediction errors, error accumulation, and sampling drift. While previous solutions have focused on modifying the sampling process (e.g., Time-Shift Samplers, Epsilon Scaling) or applying input perturbations during training, this work proposes a novel training procedure to directly mitigate this issue.

Methodology: MixFlow

The proposed MixFlow approach is motivated by the observation of a "Slow Flow" phenomenon. Empirical analysis reveals that the ground-truth noisy data nearest to the generated noisy data at a sampling timestep $t$ often corresponds to a higher-noise timestep (a "slowed timestep," $m_t \leq t$ ). In other words, the generated data lags behind the ground truth trajectory.

To leverage this observation, MixFlow modifies the training procedure as follows:

Slowed Interpolation Mixture: Instead of training the prediction network $u_\theta$ on a single interpolation $x_t$ corresponding to the training timestep $t$ , MixFlow uses a mixture of interpolations from a range of "slowed" timesteps.
- The input noisy data is sampled from a set $\mathcal{X}_t = \{x_{m_t} \mid m_t \in [(1-\gamma)t, t]\}$ , where $m_t$ represents a higher-noise (slower) timestep.
- The range size is linear with respect to $t$ , controlled by a coefficient $\gamma$ .
Modified Loss Function: The training objective optimizes the prediction network using the slowed interpolation $x_{m_t}$ as input, while the target remains the ground-truth velocity at the slowed timestep $m_t$ :
$\mathbb{E}_{t, x_0, x_1, m_t} [\| u_\theta(x_{m_t}, t) - u^*(x_{m_t}, m_t) \|_2^2]$
Crucially, the input timestep passed to the network remains the training timestep $t$ , not $m_t$ .
Sampling Strategy:
- Slowed Timestep ( $m_t$ ): Sampled uniformly from the range $[(1-\gamma)t, t]$ .
- Training Timestep ( $t$ ): To ensure that pairs of $(x_{m_t}, t)$ are sampled uniformly across all possible combinations, the training timestep $t$ is sampled from a Beta(2, 1) distribution ( $p(t) \propto t$ ), rather than a uniform distribution.
Implementation: The method is lightweight, requiring only minor code modifications (e.g., 5 lines for RAE) to existing training pipelines. It functions as a post-training strategy for pre-trained models.

Key Contributions

Identification of Slow Flow: The paper empirically demonstrates that generated noisy data at timestep $t$ is closest to ground-truth data at a slower (higher-noise) timestep, providing a theoretical basis for the proposed training modification.
MixFlow Training Algorithm: A novel training scheme that utilizes a mixture of slowed interpolations to align the training distribution more closely with the testing distribution, effectively alleviating exposure bias.
State-of-the-Art Performance: The approach achieves significant improvements across various model architectures (SiT, REPA, RAE) and generation tasks (class-conditional and text-to-image).

Experimental Results

The authors evaluated MixFlow on class-conditional image generation (ImageNet) and text-to-image generation (Stable Diffusion 3.5).

Class-Conditional Generation (ImageNet)

RAE Models: MixFlow applied to RAE models achieved new state-of-the-art results:
- 256 × 256: 1.43 FID (without guidance) and 1.10 FID (with guidance).
- 512 × 512: 1.55 FID (without guidance) and 1.10 FID (with guidance).
SiT and REPA: Consistent improvements were observed across SiT (B and XL sizes) and REPA models. For instance, SiT-B/2 with guidance improved from 4.46 to 3.91 gFID.
Sampling Steps: The gains from MixFlow were more pronounced with fewer sampling steps (e.g., 20 steps), suggesting it effectively addresses the approximation errors inherent in coarse sampling.
Comparison: MixFlow outperformed other exposure bias alleviation methods, including Input Perturbation, Epsilon Scaling, and Time-Shift Samplers.

Text-to-Image Generation

Applied to Stable Diffusion 3.5, MixFlow improved performance across all evaluated metrics (GenEval, DPG-Bench, T2I-CompBench) for both 40 and 10 sampling steps.
Qualitative results showed improvements in counting accuracy, spatial relationships, and object shape adherence.

Significance and Claims

The paper claims that MixFlow offers a simple yet effective solution to the exposure bias problem by modifying the training distribution to account for the "Slow Flow" phenomenon. The authors emphasize that:

Training vs. Inference: Unlike inference-time adjustments, MixFlow addresses the root cause by altering the training objective, making the model robust to the distribution shift between training and testing.
Compatibility: The method is compatible with various backbone architectures (SiT, DiT), interpolation schemes (linear, variance-preserving), and advanced training paradigms (REPA, RAE).
Efficiency: It achieves state-of-the-art results with minimal computational overhead, requiring only a few lines of code modification and acting as an efficient post-training step.

The authors conclude that MixFlow significantly enhances generation quality, particularly in high-resolution and low-step sampling scenarios, setting new benchmarks for diffusion-based image generation.