🤖 AI

DUET: A Diversity-Quality Duet of Distillation Experts for Two-Step Video Generation

The paper introduces DUET, a two-step video generation framework that reconciles the quality-diversity trade-off by employing a noise-level duet of independently trained experts—an sCM expert for high-noise structural diversity and a DMD expert for low-noise appearance refinement—further enhanced by DUET+ through RL-guided adaptation to achieve superior performance.

Original authors: Zian Li, Litong Gong, Borui Liao, Pengfei Liu, Xinyu Wang, Xinyuan Wei, Yifan Gao, Tiezheng Ge, Muhan Zhang

Published 2026-08-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Zian Li, Litong Gong, Borui Liao, Pengfei Liu, Xinyu Wang, Xinyuan Wei, Yifan Gao, Tiezheng Ge, Muhan Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to paint a masterpiece. In the world of artificial intelligence, there are special "diffusion models" that act like these robots. They start with a canvas full of static noise—like a TV screen with no signal—and slowly, step-by-step, they wipe away the noise to reveal a clear picture or video. The problem is, this process is incredibly slow. To get a perfect image, the robot might need to take hundreds of tiny, careful steps, which can take minutes or even hours on powerful computers. That's too slow for making movies or games in real-time.

To fix this, scientists have been trying to teach these robots to take "shortcuts." Instead of walking the whole path, they want the robot to jump straight to the finish line in just a couple of giant leaps. However, there's a catch. When you force a robot to take shortcuts, it usually has to choose between two things: making the picture look really good (sharp and detailed) or making sure every time you ask for a picture, it looks different (diverse and creative). Usually, if you get a super-sharp picture, it looks the same every time. If you get a super-creative picture, it might look a bit blurry or messy. It's like a chef who can either cook a perfect, identical steak every time, or a wildly creative, unique dish every time, but rarely both at once.

This is the puzzle a new paper called "DUET" tries to solve. The researchers, working with a video generator called Wan2.1, wanted to see if they could get the best of both worlds: a video that is both crystal clear and wildly creative, all in just two steps. They discovered that trying to mix the two cooking styles together in one pot just made a mess. Instead, they built a team of two specialists who work together like a relay race.

The Two-Step Relay Race

The paper introduces a method called DUET (Diversity–Quality Expert Tandem). Think of the video generation process as a long journey from a foggy, chaotic world (noise) to a clear, sunny world (the final video). The researchers realized that different parts of this journey need different kinds of help.

In the beginning of the journey, when the image is still very foggy, the most important thing is to decide the big picture: Where is the dog? Is it running or sleeping? Is the sun setting or rising? This is the "structure" of the video. The paper found that one type of shortcut method, called sCM, is amazing at this. It's like a rough sketch artist who can quickly draw many different, creative layouts. It's great at diversity but sometimes a bit blurry.

In the second half of the journey, when the image is already mostly clear, the most important thing is to fix the details: The texture of the fur, the reflection in the eye, the crispness of the leaves. This is the "quality" of the video. Another type of shortcut method, called DMD, is a master at this. It's like a hyper-focused editor who makes everything look sharp and perfect, but it tends to make everything look the same, losing the creative variety.

Previous attempts tried to force one robot to do both jobs at once, or to mix the two methods together. The paper argues this doesn't work well because the two methods want to do opposite things. Instead, DUET splits the work. It uses the sCM expert for the first step (the high-noise part) to lay down a diverse, creative structure. Then, it hands the baton to the DMD expert for the second step (the low-noise part) to sharpen the details and make it look professional.

The Relay and the Coach

The paper shows that this simple "relay" works surprisingly well. By letting the sCM expert handle the messy beginning and the DMD expert handle the clean ending, they get videos that are about twice as diverse as the sharp-only method, while keeping almost the same high quality. It's like having a creative director who sketches wild ideas, followed by a perfectionist editor who polishes them, without them ever arguing.

However, the researchers noticed that even this team wasn't perfect. Sometimes the hand-off between the two experts was a little clumsy, and the creative expert didn't always pick the best creative ideas. To fix this, they added a "coach" using a technique called RL-guided expert adaptation (which creates a version called DUET+).

This coach uses a reward system (like a video game scoring points for good visuals) to teach the sCM expert to aim for even better, more pleasing structures. It also helps the DMD expert get used to the specific style of sketches the sCM expert is sending over. The result, DUET+, is even better: it keeps the huge diversity advantage but pushes the quality up to match the sharpest methods available.

What the Paper Says and Doesn't Say

The authors are very clear about what they found and what they didn't. They explicitly argue against the idea that you can just mix the two methods together in a single training process. They show that trying to combine the "loss functions" (the math rules the robot follows) leads to a compromise where you lose a bit of quality and a bit of diversity, rather than getting both. They also rule out the idea that simply starting with a good sketch and then refining it (a common method called "init-then-DMD") is enough, because that approach tends to crush the diversity very quickly.

The paper proves that their "noise-level expert duet" works on a specific model called Wan2.1-T2V-1.3B. They measured this with numbers: their method achieved a diversity score about 109% higher than the sharp-only method, while keeping the quality score nearly identical. They are confident that this approach solves the trade-off problem for two-step video generation. However, they admit they only tested it on this one model size and suggest that testing it on even larger models is a job for the future. They don't claim this is the final answer for all AI video generation forever, but they do show that splitting the job by "noise level" is a simple, effective way to get both beauty and variety in a flash.

Technical Summary: DUET – A Diversity–Quality Duet of Distillation Experts for Two-Step Video Generation

1. Problem Statement

Recent advances in diffusion models have enabled high-quality video generation, but their reliance on iterative sampling incurs prohibitive computational costs, hindering practical deployment. Few-step distillation addresses this by compressing generation trajectories into a small number of network evaluations (NFEs), typically two. However, existing distillation paradigms face a fundamental quality–diversity trade-off:

Trajectory-level distillation (e.g., sCM): Preserves the teacher's sample diversity and structural layout but often yields conservative, blurry outputs due to its "mean-seeking" nature.
Distribution-level distillation (e.g., DMD): Produces sharp, high-fidelity details by matching the teacher's distribution but suffers from severe diversity collapse (mode-seeking behavior), concentrating probability mass on fewer modes.

Common attempts to resolve this by combining the two objectives into a single loss function (e.g., rCM) typically result in a compromise rather than a true solution, introducing optimization difficulties due to conflicting gradients and failing to fully inherit the strengths of either paradigm.

2. Methodology

The authors propose DUET (Diversity–qUality Expert Tandem), a framework that reconciles these paradigms through noise-level expert specialization rather than loss-level combination.

Core Architecture: Noise-Level Expert Duet

DUET assigns distinct roles to two independently trained experts based on the noise level of the sampling schedule:

High-Noise Interval ( $t \in [1, \tau]$ ): Assigned to an sCM expert. This stage determines coarse variables such as object identity, layout, composition, and motion patterns. The sCM expert is trained with its native consistency objective to ensure broad coverage of the sample space and structural diversity.
Low-Noise Interval ( $t \in [\tau, 0]$ ): Assigned to a DMD expert. This stage refines visual details, textures, and appearance. The DMD expert is trained with its native distribution-matching objective to ensure high fidelity and sharpness.

Inference Process:
The two experts operate in a relay manner within a single two-step generation process:

The sCM expert predicts a clean endpoint from the initial noise ( $x_1$ ).
This prediction is re-noised to the switch time $\tau$ using a re-noising operator.
The DMD expert takes the re-noised latent at $\tau$ and performs the final step to generate the clean video latent ( $\hat{x}_0$ ).
The switch time $\tau$ is determined by analyzing the curvature of the teacher's flow-matching ODE trajectory; empirically, $\tau = 0.8$ is chosen as the transition point where the trajectory shifts from rapid structural formation to slow detail refinement.

DUET+: RL-Guided Expert Adaptation

To address remaining bottlenecks—specifically the distribution gap between the sCM expert's output and the DMD expert's training distribution, and the capacity limits of the sCM expert—the authors introduce DUET+. This involves a role-aware adaptation strategy:

sCM Expert Adaptation: The sCM expert is fine-tuned using GRPO (Group Relative Policy Optimization). This steers the diverse outputs of the sCM expert toward higher-reward structures (based on human preference scores like HPSv3) while preserving its coverage behavior.
DMD Expert Adaptation: The DMD expert continues training with its native loss but uses the actual intermediate latents produced by the adapted sCM expert during inference, rather than the teacher's native backward simulation. This aligns the DMD expert with the true relay interface.

3. Key Contributions

Objective-Level Diagnosis: The paper identifies that the quality–diversity trade-off stems from the intrinsic conflict between trajectory-level and distribution-level objectives, which induce distinct noise-to-data mappings. Forcing a single set of parameters to satisfy both leads to optimization hardness.
Noise-Level Expert Duet (DUET): A novel architecture that decouples the objectives by assigning them to specific noise regimes. This sidesteps gradient conflicts and loss-weight tuning, allowing the model to jointly achieve diversity and quality.
Role-Aware Adaptation (DUET+): A lightweight RL-guided adaptation that exploits the sCM expert's diversity for preference optimization and repairs the DMD expert's relay interface, further improving overall quality without sacrificing diversity.
Empirical Validation: Demonstrating that in the extreme two-step regime, it is possible to approach DMD-level quality while retaining nearly all of sCM's structural diversity.

4. Experimental Results

Experiments were conducted using the Wan2.1-T2V-1.3B backbone for text-to-video generation at 832 $\times$ 480 resolution.

Quality vs. Diversity:
- sCM: Achieves the highest diversity (ViCLIP average: 0.1865) but lower quality (VBench average: 81.51).
- DMD: Achieves high quality (VBench average: 84.38) but low diversity (ViCLIP average: 0.0727).
- rCM (Baseline): Shows a trade-off; even with tuning, its diversity remains significantly lower than sCM.
- DUET: Achieves a quality score of 83.96 (close to DMD) while maintaining a diversity score of 0.1512 (a 108.0% increase over DMD and retaining much of sCM's structural variety).
- DUET+: Further improves quality to 84.40 while preserving the diversity advantage (ViCLIP: 0.1521).
Qualitative Analysis:
- DUET preserves diverse layouts and poses (from sCM) while sharpening details (from DMD).
- In style-specific prompts (e.g., pixel art), DUET retains lower-probability styles that DMD often collapses into smoother, generic modes.
- DUET+ produces cleaner, more coherent frames with fewer artifacts compared to the base relay.
Ablation Studies:
- Switch Time ( $\tau$ ): $\tau=0.8$ was found optimal. Lower values overburden the sCM expert (blurry results), while higher values leave the DMD expert with insufficient refinement room.
- Reward Function: HPSv3 was selected as the reward for sCM adaptation, showing superior performance in consistency and aesthetic quality compared to SFS-Nature or TA.
- Initialization: Unlike "init-then-DMD" methods (which initialize DMD from a few-step generator and then train), DUET+ maintains distinct expert roles. "Init-then-DMD" quickly collapses inherited diversity, whereas DUET+ preserves it.

5. Significance and Claims

The paper claims that noise-level expert specialization is a simple yet effective paradigm for reconciling diversity and quality in few-step video generation. By decoupling the optimization of structural diversity and visual fidelity into separate experts operating at different noise levels, DUET avoids the "compromise" inherent in loss-level combinations.

The authors emphasize that their results establish a new standard for two-step generation, showing that high quality and high diversity are not mutually exclusive if the distillation process is structured to leverage the complementary strengths of different objectives. While the current evaluation is limited to the Wan2.1-1.3B model, the authors note that the underlying properties of sCM and DMD are not backbone-specific, suggesting potential for scaling to larger models in future work.