Technical Summary: Making Time Editable in Video Diffusion Transformers

Problem Statement

Modern Diffusion Transformers (DiTs) have achieved significant success in generating high-fidelity video frames and realistic motion. However, they suffer from limited temporal controllability. In standard DiT-based video generation, "diffusion time" acts as an overloaded variable, serving as a proxy for both the progress of denoising and the evolution of motion. Consequently, temporal dynamics are not modeled as explicit, independently controllable variables but are instead entangled with content and motion representations.

This entanglement creates ambiguity: the model struggles to distinguish how far denoising has progressed from how motion should unfold over time. As a result, when the target temporal grid (e.g., Frames Per Second or FPS) changes relative to training, the model fails to maintain consistent temporal dynamics. It may produce plausible individual frames but struggles with intermediate motion quality, leading to artifacts in articulated structures (e.g., limbs) or the failure of gradual physical processes (e.g., fog dissipation) to evolve correctly. Existing methods control what moves or where it moves, but they do not make the temporal progression itself an explicitly editable variable.

Methodology

The authors propose a methodology to extend pretrained DiT backbones with an explicit temporal control module, termed the Time Adapter (TA). This approach separates global motion pacing from local temporal alignment without redesigning the backbone architecture.

Core Architecture: Time Adapter

The method introduces two complementary temporal signals integrated via lightweight additive adapters:

1. Global FPS Conditioning: An embedding $E_{fps}(fps)$ derived from the input FPS (e.g., 15, 24, 30, 60). This modulates the global motion rate through the denoising conditioning pathway, determining the expected pacing of scene evolution.
2. Local Latent-Time Embeddings: An embedding $E_{\tau}(\tau)$ representing frame-wise temporal positions ($\tau$). This provides local temporal ordering and correspondence at the token level, aligning the progression of specific frames.

The conditioning vector $e$ is formulated as:
$$e = E_t(t) + E_{fps}(fps)$$
where $E_t(t)$ is the standard diffusion time embedding. The latent-time embedding is added to the patch embeddings:
$$h_0 = \text{PatchEmbed}(x_t) + E_{\tau}(\tau)$$

This factorization allows the model to represent temporal progression explicitly rather than relying solely on diffusion time as an implicit proxy. The adapters are initialized to preserve the pretrained backbone's behavior.

Training Regimes

The authors evaluate two training strategies:

• Time Adapter (TA): Only the temporal branches (adapters) are trained while the DiT backbone remains frozen. This preserves the original generative prior.
• Fine-Tuned Time Adapter (FTTA): The same temporal conditioning is used, but the full DiT backbone is additionally fine-tuned. This introduces stronger global temporal regularization.

The models are trained on a curated dataset of approximately 40,000 videos covering human actions and natural processes across multiple FPS regimes (24–60 FPS).

Key Contributions

1. Explicit Temporal Factorization: The paper introduces a framework that decouples time from the denoising process by separating it into global motion pacing (FPS) and local temporal alignment (latent time).
2. Backbone Preservation: The TA approach allows for explicit time editing without redesigning the backbone, retaining the visual and structural priors of the pretrained model while expanding its controllable dynamic range.
3. Cross-Architecture Transferability: The authors demonstrate that explicit temporal factorization transfers across different DiT backbones (Wan2.2 and Kandinsky 5.0 Lite) with varying parameter scales and output resolutions.
4. Improved Temporal Consistency: The method improves intermediate motion consistency and reduces perceptual artifacts in articulated dynamics, even at standard playback speeds, by exposing the model to denser temporal transitions during training.

Experimental Results

The evaluation utilizes the VMBench benchmark across two domains: Human Actions and Natural Processes, tested at 15, 24, 30, and 60 FPS.

Quantitative Findings

• Human Actions: Both TA and FTTA reduce Flow Acceleration and LPIPS-Var compared to the baseline, indicating more stable short-term transitions. TA achieves the strongest Flow Directional Consistency (FDC) at 24–60 FPS, suggesting that explicit conditioning improves intermediate motion consistency when the target FPS aligns with the training regime.
• Natural Processes: FTTA consistently achieves the strongest temporal regularization, yielding the best Motion Smoothness Scores (MSS) and lowest Flow Acceleration. This is attributed to FTTA's ability to restructure distributed physical processes (e.g., fog, waves) to be more responsive to temporal conditioning.
• Metric Limitations: The authors note that standard metrics do not always reflect improvements in temporal quality. For instance, articulated artifacts can increase Perceptible Amplitude Score (PAS) despite reducing perceived motion quality, creating a discrepancy between quantitative scores and human evaluation.

Qualitative and Human Evaluation

• Side-by-Side (SBS) Evaluation: Human raters preferred TA for articulated human motion and naturalness, while FTTA was preferred for distributed physical processes and speed consistency.
• Physical Evolution: Unlike post-hoc FPS resampling (which merely interpolates frames), the proposed method changes the generated temporal evolution itself. Increasing the target FPS results in faster scene progression rather than just smoother playback.
• Generalization: The qualitative trends observed on Wan2.2 were replicated on Kandinsky 5.0 Lite, supporting the claim that explicit temporal factorization is not architecture-specific.

Significance and Limitations

Significance

The paper argues that treating time as a single overloaded variable is a fundamental limitation in current video diffusion models. By making time editable through explicit factorization, the authors enable more physically consistent and reliable video generation. The work suggests that temporal controllability is not merely a side effect of denoising but can be a primary, controllable variable.

Limitations

The authors explicitly acknowledge three limitations:

1. Training Regime Sensitivity: While temporal factorization improves controllability within the training distribution, the model remains sensitive to underrepresented temporal scales (e.g., low FPS). Explicit time representations cannot fully compensate for missing temporal priors in the training data.
2. Fixed Generation Horizon: The method makes temporal dynamics more controllable but does not increase the number of generated frames. Strong temporal acceleration compresses the available time for action realization, potentially leading to truncated behaviors.
3. Evaluation Metrics: Current metrics struggle to capture perceptual artifacts in intermediate motion, particularly for articulated structures, leading to gaps between quantitative scores and perceived temporal quality.

The authors conclude that making time editable is a necessary step toward reliable video generation but is not sufficient for long-horizon generation, which will likely require combining explicit temporal representations with adaptive generation lengths.