🤖 machine learning

Parallel Decoding Distillation for Fast Image and Video Generation

This paper introduces Parallel Decoding Distillation (PDD), a scalable and simplified trajectory-based method that accelerates image and video generation by predicting multiple denoising steps per network evaluation, achieving state-of-the-art performance with 4–8 function evaluations while significantly improving video diversity compared to existing distillation techniques.

Original authors: Neta Shaul, Chao Liu, Arash Vahdat, Julius Berner

Published 2026-07-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Neta Shaul, Chao Liu, Arash Vahdat, Julius Berner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to paint a masterpiece or direct a movie. In the world of artificial intelligence, this is done using "diffusion" or "flow" models. Think of these models as artists who start with a canvas covered in static noise—like a TV tuned to a dead channel—and slowly, step-by-step, clean it up until a clear image or video appears. The problem is that this cleaning process is incredibly slow. To get a high-quality result, the robot might need to take hundreds of tiny steps, checking its work at every single one. It's like trying to walk across a room by taking one millimeter step at a time; you'll get there, but it will take forever.

To speed this up, scientists have been trying to teach these robots to take bigger leaps. Some methods try to guess the final destination in one giant jump, while others try to learn the "path" the robot should take so it can skip the boring middle parts. However, the current best methods for video are tricky. They often rely on complex, unstable training tricks that can make the robot forget how to move, resulting in videos that look great but are frozen in time, or they collapse into repetitive, boring patterns. The big question is: Can we teach these models to move fast and smoothly without losing the magic of their creativity?

This paper introduces a new technique called Parallel Decoding Distillation (PDD), which acts like a super-efficient coach for these AI artists. Instead of forcing the robot to take one step at a time, PDD teaches it to predict a whole sequence of steps in a single glance. Imagine you are walking down a hallway. A normal robot stops at every door to check if it's open before moving to the next. PDD, however, looks down the entire hallway, predicts exactly how the floor will feel under its feet for the next ten steps, and then strides forward confidently, covering that whole distance in one go.

The authors found that by training the model to predict the "average speed" (or velocity) for a block of time all at once, they could generate high-quality images and videos in just 4 to 8 steps (called Function Evaluations, or NFE), compared to the hundreds usually required. This is a massive speedup. Unlike previous methods that tried to force the robot to follow a rigid path or use complicated math that often broke the model's ability to create variety, PDD uses a simpler, more direct approach. It doesn't need to calculate complex derivatives or use adversarial games (where two AIs fight each other to improve), which are known to cause "mode collapse"—a fancy way of saying the AI gets stuck making the same thing over and over.

The paper shows that this method works incredibly well on large-scale models, including those that generate text-to-video and text-to-image content. For example, on the Wan2.1 video model, PDD produced videos with 4 NFE that were not only faster but also more diverse and dynamic than other top methods. In tests with the LTX-2.3 model, which creates 10-second videos with audio, PDD managed to match the quality of a teacher model that took 4 × 30 NFE (120 steps total) using only 8 NFE. The results suggest that PDD is a robust way to make AI generation faster without sacrificing the variety and motion that make videos feel alive. It's a significant step forward, proving that you don't need to take a million tiny steps to get a great result; sometimes, you just need to learn how to see the whole path ahead.

Technical Summary: Parallel Decoding Distillation for Fast Image and Video Generation

1. Problem Statement

Large-scale diffusion and flow matching models have achieved state-of-the-art (SOTA) results in text-to-image, text-to-video, and multi-modal generation. However, their inherent iterative sampling processes require hundreds of network evaluations (NFEs), creating significant computational costs and latency. This bottleneck hinders applications such as real-time video generation, content editing, and interactive world modeling.

Existing acceleration methods generally fall into two categories:

Trajectory-based methods: These distill the teacher's sequential sampling process into fewer steps. While effective for images, they often degrade video quality or require costly training algorithms (e.g., Jacobian-vector products or finite differences) when applied to video models.
Distribution-based methods: These align the marginal distributions of the student and teacher rather than following specific trajectories. While dominant in video distillation (e.g., using Variational Score Distillation or adversarial losses), they suffer from mode collapse, leading to a loss of video diversity and static motion. They also often require alternating training objectives and are sensitive to hyperparameters.

The paper identifies a need for a scalable, trajectory-based distillation method that avoids the optimization difficulties of distribution-based methods (mode collapse) and the computational overhead of existing trajectory-based methods (JVPs, finite differences), specifically for high-resolution video generation.

2. Methodology: Parallel Decoding Distillation (PDD)

The authors introduce Parallel Decoding Distillation (PDD), a trajectory-based distillation framework designed to accelerate inference for diffusion and flow matching models.

Core Concept

Unlike traditional methods that merge multiple denoising steps into a single large step, PDD learns a parallel decoder that predicts the mean velocities for multiple consecutive time intervals in a single network evaluation.

Discretization: The time domain $[0, 1]$ is discretized into $N$ intervals. These are grouped into blocks of size $L$ .
Parallel Prediction: Given a state $X_n$ at time $t_n$ , the parallel decoder $\bar{u}_\theta$ predicts the mean velocities for all $L$ intervals within the block $\{n, \dots, n+L-1\}$ simultaneously:
$\bar{u}_\theta(n|X_n) \approx u_k(X_k), \quad k = n, \dots, n+L-1$
where $u_k$ is the mean velocity of the $k$ -th interval.
Sampling: During inference, the model advances $L$ intervals in one step using the predicted velocities, reducing the total NFEs to $N/L$ . By varying the block size $L$ during training, the model supports variable NFEs at inference time without additional time conditioning.

Training Objective

PDD employs a simplified, regression-based training objective that avoids VSD, adversarial losses, JVPs, and finite differences.

Teacher Approximation: The target mean velocities are approximated using a Runge-Kutta solver (Euler or Midpoint) applied to the pre-trained teacher model.
On-Policy Training: The loss is computed on the student's own outputs (on-policy). The student predicts velocities, simulates the trajectory within the block, and the loss is calculated by comparing the student's predicted velocity at a sampled step $k$ against the teacher's approximated mean velocity at that state.
Loss Function:
$\mathcal{L}_{PD}(\theta) = \mathbb{E} \left[ \| \bar{u}_\theta(n|X_n) - u_k(\text{sg}(\bar{X}_k)) \|^2 \right]$
where $\text{sg}$ denotes the stop-gradient operator, and $\bar{X}_k$ is the state reached by following the student's predicted velocities.

Architecture

Backbone: PDD utilizes the exact same backbone architecture as the pre-trained teacher model.
Output Layer: The final linear layer is expanded by repeating the channel dimension $N$ times (where $N$ is the grid size). This allows the model to output $N$ distinct velocity predictions corresponding to each time step.
Layer Fusion: During generation, instead of applying $L$ separate linear layers, the weights are fused into a single linear layer that outputs the weighted average velocity for the block. This ensures no additional computational cost during inference compared to the teacher's single-step evaluation.

Data-Free Training

For large-scale models where data is unavailable, PDD supports a data-free training scheme. The model samples an initial noise, alternates between solver steps (advancing the state using the student's own predictions) and training steps, effectively learning from its own generated trajectories.

3. Key Contributions

Formulation of PDD: A scalable, trajectory-based distillation method that predicts mean velocities for multiple intervals in a single evaluation, enabling fast inference for flow matching and diffusion models.
Simplified Training: A single regression-based objective that eliminates the need for JVPs, finite differences, multi-stage training, VSD, or GAN losses, while still producing high-quality and diverse samples.
Flexible Architecture: A design compatible with any pre-trained model that supports variable NFEs at inference time without requiring additional time conditioning or a second time coordinate.
Empirical Validation: Demonstration of SOTA performance on ImageNet-256, Qwen-Image (Text-to-Image), Wan2.1 (Text-to-Video 1.3B/14B), and LTX-2.3 (Text-to-Video/Audio), with significant improvements in generation diversity compared to distribution-based baselines.

4. Experimental Results

The authors evaluated PDD across three tasks:

Class-Conditional Image Generation (ImageNet-256):
- PDD achieved competitive FID scores in the single-step setting (FID 2.69 with Midpoint), outperforming or matching SOTA methods like FreeFlow and Pi-Flow, while supporting variable NFEs (1, 2, 4, 8).
- The method showed stable convergence across different hyperparameters.
Text-to-Image (Qwen-Image 20B):
- Evaluated on OneIG, DPG-Bench, and GenEval benchmarks with 2, 4, and 8 NFEs.
- PDD (Midpoint) achieved the best overall scores on OneIG and DPG-Bench at 4 and 8 NFEs, outperforming baselines like DMD2, TwinFlow, and Pi-Flow.
- Crucially, PDD preserved significantly higher diversity (measured by OneIG diversity metrics) compared to DMD2, which suffered from mode collapse.
Text-to-Video (Wan2.1 1.3B & 14B):
- Evaluated on VBench with 4 and 8 NFEs.
- PDD achieved SOTA video quality and overall scores, ranking first on the 1.3B model and second on the 14B model (behind AnyFlow in overall metrics but with better diversity).
- Diversity: PDD demonstrated superior diversity scores compared to distribution-based baselines (DMD2, AnyFlow).
- Motion: Visual analysis indicated that PDD-generated videos exhibited higher degrees of motion and dynamic consistency compared to baselines, which often produced static or repetitive videos.
Text-to-Video/Audio (LTX-2.3):
- PDD distilled the 22B LTX-2.3 model for 10s 720p video generation with audio using only 8 NFEs.
- After just 250 training iterations, PDD performed on par or better than the official 8-step distilled model, despite having no access to training data.
- Human preference evaluation (via Gemini 3.1 Pro) showed PDD winning or tying the official distilled model in prompt alignment, visual quality, motion quality, and audio quality.

5. Significance and Claims

The paper claims that PDD represents the first pure trajectory-based distillation method capable of achieving high-resolution, few-step video generation with improved diversity.

Overcoming Mode Collapse: By adhering to a trajectory-based approach without relying on distribution-based losses (VSD/GANs), PDD avoids the mode collapse and lack of motion often seen in current video distillation methods.
Computational Efficiency: The method removes the need for expensive JVPs and finite differences, making it scalable to large models (up to 22B parameters) and compatible with data-free training.
Flexibility: The ability to support variable NFEs at inference time via block-size selection offers a practical advantage for diverse application scenarios.
Motion and Diversity: The authors emphasize that PDD not only accelerates generation but also preserves the "richness" of the teacher model's output, specifically regarding motion dynamics and sample diversity, which are critical for video generation.

The authors conclude that while PDD is currently limited to flow and diffusion models, the principle of parallel decoding could potentially be generalized to discrete autoregressive models in future work. They also note that investigating PDD in data-dependent settings beyond ImageNet-256 remains an open area for future research.