Technical Summary: PUSA V1.0

Problem Statement

Current video diffusion models (VDMs) predominantly rely on a scalar timestep variable to control the noise level and evolution trajectory uniformly across all frames in a video clip. While effective for Text-to-Video (T2V) generation, this rigid synchronization imposes fundamental limitations on temporal modeling. Specifically, it struggles with nuanced, temporal-dependent tasks such as Image-to-Video (I2V), video extension, and start-end frame conditioning. Existing solutions to these challenges often involve:

• Destructive Adaptation: Fine-tuning base models with massive resources and task-specific architectures (e.g., mask mechanisms), which risks catastrophic forgetting of the base model's generative priors.
• Autoregressive Approaches: Models that generate frames sequentially to avoid rigid synchronization. However, these are often constrained by computational inefficiency, error accumulation over long sequences, and an inability to perform bidirectional reasoning (e.g., conditioning on both start and end frames).

Methodology

The paper introduces PUSA V1.0, a versatile model that leverages Vectorized Timestep Adaptation (VTA) to enable fine-grained temporal control within a unified video diffusion framework. The core innovation is a non-destructive adaptation strategy that modifies the base model's temporal conditioning without altering its architecture or destroying its pretrained capabilities.

1. Vectorized Timestep Adaptation (VTA)

Instead of a single scalar timestep $t$, PUSA employs a vectorized timestep $\tau \in [0, 1]^N$, where $N$ is the number of frames. Each component $\tau^i$ represents the individual progression parameter of the $i$-th frame along its probability path.

• Architectural Modification: The adaptation is minimal and non-destructive. The timestep embedding module is re-engineered to process the input vector $\tau$, generating frame-specific embeddings. These embeddings are projected to produce per-frame modulation parameters (scale, shift, gate) within the DiT (Diffusion Transformer) blocks.
• Preservation of Priors: Because the modification is non-destructive, setting all frame timesteps to the same value allows the model to generate results identical to the base T2V model, fully preserving its original capabilities.

2. Frame-Aware Flow Matching (FAFM)

The model is trained using a Flow Matching objective extended for vectorized timesteps.

• Training Strategy: The training utilizes a fully randomized vectorized timestep strategy ($p_{async} = 1$), where each component $\tau^i$ is sampled independently from a uniform distribution $U[0, 1]$. This compels the model to learn fine-grained temporal control from a maximally diverse distribution of temporal states.
• Efficiency: Unlike previous methods that require complex sampling schedules or massive datasets, this approach allows the model to be fine-tuned with extremely limited data and compute.

3. Inference for Zero-Shot Tasks

PUSA achieves zero-shot capabilities by strategically manipulating the vectorized timestep $\tau$ during sampling:

• I2V: The timestep component for the conditioning image (e.g., the first frame) is clamped to zero ($\tau^1 = 0$) throughout inference, effectively fixing it as a clean condition while other frames evolve.
• Start-End Frames & Extension: By clamping timesteps for specific subsets of frames (e.g., first and last frames) to zero or specific noise levels, the model can generate coherent videos conditioned on arbitrary frame configurations without task-specific retraining.

Key Contributions

1. Unprecedented Efficiency: PUSA achieves State-of-the-Art (SOTA) I2V performance using only 4,000 samples and 0.5K compute cost (approx. $500), compared to the expected >10M samples and >100K cost for comparable baselines like Wan-I2V.
2. Unified Multi-Task Generalization: The model retains the base model's T2V capabilities while simultaneously generalizing to I2V, start-end frames, video extension, and completion in a zero-shot manner.
3. Mechanistic Insight: Analyses reveal that the approach surgically injects temporal dynamics into the model's self-attention mechanisms while preserving the foundation model's generative priors. This avoids the "combinatorial explosion" inherent in vectorized timesteps by leveraging the base model's robustness to asynchronous evolution.

Results

• Quantitative Performance: On the VBench-I2V benchmark, PUSA achieves a total score of 87.32, outperforming Wan-I2V (86.86) and other open-source models. It demonstrates superior performance in I2V Subject Consistency (97.64 vs. 96.95) and Background Consistency (99.24 vs. 96.44).
• Training Efficiency: Ablation studies show that PUSA converges rapidly (within 900 iterations) and significantly outperforms baselines trained with the same budget. The LoRA-based fine-tuning approach proves particularly effective, achieving higher scores than full fine-tuning at the same iteration count.
• Qualitative Analysis: Visualizations of attention maps show that while the base T2V model exhibits diagonal attention (frame independence) and Wan-I2V shows transient attention to the first frame, PUSA maintains strong, consistent attention to the conditioning frame across all inference steps. Parameter divergence analysis confirms that PUSA induces minimal changes (mostly in self-attention blocks) compared to the substantial drift seen in Wan-I2V.

Significance and Claims

The paper positions PUSA as a critical shift in video generation paradigms. By combining principled temporal modeling (vectorized timesteps) with efficient, non-destructive adaptation, PUSA redefines the efficiency-quality tradeoff.

• Democratization: The method democratizes high-fidelity video generation by making SOTA-level multi-task synthesis accessible with minimal computational resources.
• Versatility: It establishes a scalable paradigm that unifies T2V, I2V, and complex temporal editing tasks within a single framework, overcoming the limitations of both rigid scalar-timestep models and error-prone autoregressive approaches.
• Non-Destructive Nature: The work demonstrates that advanced temporal control can be unlocked without sacrificing the foundational capabilities of large-scale pre-trained models, avoiding the pitfalls of catastrophic forgetting.