🤖 machine learning

DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Models

The paper proposes DreOPD, a novel on-policy distillation method for flow-matching models that converts implicit reward extrapolation into closed-form velocity regression and leverages a degraded reference to achieve stable, high-performance post-training that outperforms both standard on-policy distillation and multi-task reinforcement learning baselines.

Original authors: Mingfeng Lin, Chengfei Cai, Lin Xu, Yuxiang Wei, Liang Han

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mingfeng Lin, Chengfei Cai, Lin Xu, Yuxiang Wei, Liang Han

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers are learning to paint, write, and dream. For a long time, scientists have been teaching these machines using a method called "Flow-Matching." Think of this like a sculptor starting with a block of noisy, static-filled marble and slowly chipping away the chaos until a perfect statue emerges. The computer learns to predict exactly how to remove the noise at every single step to create a beautiful image from a text description. But here's the catch: while these digital sculptors are great at making something, they aren't always great at making exactly what you want. They might struggle to follow complex instructions, write the right words inside the picture, or make the scene look aesthetically pleasing.

To fix this, researchers usually try two main tricks. The first is "Reinforcement Learning," which is like hiring a strict art critic who gives the computer a score after every attempt. The computer tries again and again, hoping to get a higher score. The problem? This is a chaotic process. The computer might get confused by the feedback, or if you ask it to learn three different skills at once (like writing text, drawing animals, and looking pretty), the instructions can clash, and the computer gets stuck. The second trick is "Distillation," which is like having a master painter (the Teacher) stand next to the student and say, "Do exactly what I do." This is stable and safe, but the student can never become better than the teacher; they just become a copy. The big question scientists are asking is: Can we teach a student to not only copy the master but to actually surpass them, especially when we have multiple masters with different specialties?

This paper introduces a clever new method called DreOPD (Degraded-reference Extrapolative On-policy Distillation) to solve this puzzle. The researchers found a way to combine the stability of copying a teacher with the ambition of beating them. Instead of just telling the student to mimic the teacher's brushstrokes, DreOPD gives the student a slightly "imperfect" or "degraded" version of the teacher to compare against. Imagine a student looking at a master painter's work, but also looking at a slightly blurry, low-quality photocopy of that same painting. By measuring the difference between the blurry copy and the sharp original, the student can figure out exactly which direction to push their own painting to make it even better than the original.

The paper suggests that by using this "degraded reference," the computer can calculate a precise mathematical path to move beyond the teacher's capabilities. It's like having a GPS that doesn't just tell you where the teacher is, but points in the direction away from a bad version of the teacher, guiding the student straight toward a new, improved destination. The researchers tested this on a powerful image generator (SD3.5-M) and found that their method allowed the student model to outperform the specialized teachers in most areas. For example, in tasks like counting objects correctly (GenEval) or writing text inside images (OCR), the DreOPD student scored higher than the experts it was trained on.

The paper explicitly rules out the idea that simply copying a teacher is enough to get the best results, and it also argues that standard Reinforcement Learning is often too unstable and prone to errors when trying to learn multiple skills at once. Instead, the authors suggest that their "extrapolation" approach—using the gap between a good teacher and a slightly worse reference—is the key. They measured this with specific scores: for instance, on a text-rendering test, their method improved the score from 0.9239 to 0.9364, beating the teacher. They also showed that if the "degraded reference" is too weak, the student doesn't get enough push, but if it's too messy, the student gets confused. The sweet spot, they found, was a mild degradation, like an 8-bit quantization of the teacher's output, which provided just the right amount of contrast to guide the student to new heights.

In short, DreOPD is a new way to train AI artists that turns the concept of "copying" into "surpassing." By using a slightly flawed version of the teacher as a stepping stone, the student learns to leap further than the teacher ever could, creating images that are not just good, but better than the sum of their parts. The authors suggest this could be a game-changer for making AI that is more reliable, follows instructions better, and creates more beautiful art, all without the chaotic instability of traditional training methods.

Technical Summary: DreOPD

Problem Statement

Flow-matching models have become a mainstream paradigm for text-to-image generation. However, adapting these models to diverse downstream objectives—such as prompt following, text rendering, aesthetics, and human preference—typically requires post-training. This adaptation faces two primary challenges:

Conflict of Objectives: Task-specific optimization objectives often exhibit implicit conflicts, making it difficult for a single model to improve multiple capabilities simultaneously.
Limitations of Existing Post-Training Methods:
- Reinforcement Learning (RL): While RL enables direct optimization of task-specific rewards and potential improvement beyond existing models, trajectory-level optimization often suffers from high-variance gradients, reward exploitation, and interference among heterogeneous objectives.
- On-Policy Distillation (OPD): OPD offers dense and stable supervision by matching student rollouts to teacher predictions. However, conventional OPD is imitation-based; the student interpolates among specialized teachers rather than systematically exceeding them.

A practical question arises: How can a student consolidate multiple strong teachers, leverage readily available weaker models (e.g., early checkpoints or degraded versions), and improve beyond the teachers themselves?

Methodology: DreOPD

The authors propose DreOPD (Degraded-reference extrapolative On-Policy Distillation), a method that bridges the gap between stable imitation and reward extrapolation for flow-matching models.

1. From Implicit Reward Extrapolation to Closed-Form Velocity Targets

The core theoretical contribution is translating the distribution-level principle of reward extrapolation (previously established for autoregressive language models) into the native velocity-regression objective of flow-matching.

Theoretical Derivation: Under the assumption of shared-covariance Gaussian transitions, the authors derive a closed-form optimizer for the conditional transition objective.
The Target: The optimal velocity target $v^\star_\lambda$ $v_{λ}^{⋆}$ at a student-visited state is given by:
$v^\star_\lambda = v_T + (\lambda - 1)(v_T - v_{ref})$
Where $v_T$ $v_{T}$ is the teacher velocity, $v_{ref}$ $v_{r e f}$ is the reference velocity, and $\lambda \ge 1$ $λ \geq 1$ is the extrapolation strength.
- At $\lambda = 1$ , the target reduces to teacher imitation ( $v_T$ ).
- At $\lambda > 1$ , the target moves beyond the teacher along the direction defined by the teacher-reference contrast.
Training: The student is trained via on-policy regression, minimizing the distance between its predicted velocity and this extrapolated target $v^\star_\lambda$ on its own generated trajectories.

2. Reward Interpretation and Conditions

The paper characterizes conditions under which this geometric extrapolation aligns with reward maximization.

Reward-Tilt Assumption: If the teacher is a solution to a KL-regularized reward optimization problem (i.e., $p_T \propto p_{ref} \exp(r/\beta)$ ), the teacher-reference log-density ratio recovers the reward signal.
Monotonicity: Under this assumption, the expected reward of the extrapolated distribution increases monotonically with $\lambda$ , provided the reward is non-constant. This justifies extrapolating beyond the teacher.

3. Amplifying Extrapolation with a Degraded Reference

The magnitude of extrapolation is determined by the vector $(v_T - v_{ref})$ . A reference close to the teacher yields a weak signal.

Strategy: The authors introduce a mildly degraded reference ( $v_{deg}$ ) to enlarge the teacher-reference contrast while preserving the generator's structural integrity.
Implementation: Degradation is achieved via controlled corruption, such as weight quantization (e.g., 8-bit) or mild noise injection into the velocity output of the base model.
Theoretical Justification: Proposition 3 demonstrates that within a reward-alignment model, using a reference with lower reward alignment ( $\rho_d < \rho_n < 1$ ) increases the effective reward-tilt coefficient and the local rate of reward change without altering the reward direction.

4. Training Procedure

The training loop (Algorithm 1) iterates over tasks:

Sample prompts for a specific task.
Rollout the current student to generate trajectories.
Query the task-specific teacher and the shared degraded reference at visited states.
Compute the closed-form extrapolated target $v^\star_\lambda$ .
Update the student via regression toward this target.

Key Contributions

Closed-Form Target Derivation: The authors derive a closed-form velocity target that extends flow-based OPD from simple teacher imitation to reward extrapolation.
Characterization of Contrast: They characterize the conditions under which teacher-reference contrast is reward-aligned and introduce controlled reference degradation to strengthen this contrast.
Empirical Superiority: Across single- and multi-teacher settings, DreOPD achieves the best average performance among baselines and surpasses specialized teachers on most metrics.

Experimental Results

Experiments were conducted using SD3.5-M on three tasks: GenEval (compositional prompt following), OCR (text rendering), and Aesthetics (human preference).

Single-Teacher Distillation

DreOPD outperformed standard OPD baselines (Flow-OPD, DiffusionOPD) and the specialized teachers themselves.
Metrics: It improved GenEval from 0.9470 (Teacher) to 0.9710, OCR from 0.9239 to 0.9364, and HPSv2.1 from 0.3460 to 0.3497.
The use of a degraded reference provided consistent gains over the non-degraded variant.

Multi-Teacher Distillation

In a setting consolidating three specialized teachers, DreOPD achieved the highest average score (0.9939) compared to multi-task RL baselines (Flow-GRPO, GRPO-Guard, DiffusionNFT) and other OPD methods.
It surpassed the corresponding teachers on GenEval, OCR, HPSv2.1, Aesthetic, and ImageReward metrics, while matching or retaining performance on PickScore and ClipScore.
Ablation studies confirmed that an extrapolation strength of $\lambda = 1.25$ provided the best cross-task balance and that 8-bit velocity quantization was the most effective degradation strategy.

Significance and Claims

The paper claims that DreOPD establishes reward extrapolation as an effective framework for consolidating specialized flow models. By converting implicit reward extrapolation into a closed-form velocity regression, the method retains the stability of on-policy distillation while enabling the student to systematically exceed the performance of its teachers. The introduction of a degraded reference is highlighted as a practical mechanism to strengthen the extrapolation signal without disrupting the generative structure. The results suggest that a single model can be trained to perform well across diverse dimensions, overcoming the trade-offs typically found in multi-task RL or simple teacher interpolation.