💻 computer science

UniTemp: Unlocking Video Generation in Any Temporal Order via Bidirectional Distillation

UniTemp is a bidirectional distillation framework that enables autoregressive video diffusion models to generate videos in arbitrary temporal orders by overcoming the limitations of causal 3D VAEs through the introduction of blockwise anchor latents.

Original authors: Lin Zhang, Sicheng Mo, Zefan Cai, Jinhong Lin, Zihao Lin, Jiuxiang Gu, Krishna Kumar Singh, Yuheng Li, Yin Li

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lin Zhang, Sicheng Mo, Zefan Cai, Jinhong Lin, Zihao Lin, Jiuxiang Gu, Krishna Kumar Singh, Yuheng Li, Yin Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to write a movie script. Most AI video generators today are like writers who can only write a story from start to finish. They see the first scene, write the second, then the third, and so on. They are great at this, but they are stuck in a "forward-only" mindset.

If you wanted to write a "prequel" (what happened before the first scene) or fill in a gap between two existing scenes, these writers would get confused and the story would fall apart.

UniTemp is a new AI system that breaks this rule. It allows the AI to write a video story in any order: forward, backward, or filling in the middle. Here is how it works, explained simply:

The Problem: The "One-Way Street"

The paper explains that current video AI uses a special "translator" (called a Causal 3D VAE) to turn video into data the computer can understand.

The Analogy: Imagine this translator is like a person reading a book who can only see the pages before the current one. They know what happened in Chapter 1 when they are reading Chapter 2.
The Issue: If you try to write the story backward (starting with Chapter 10 and working back to Chapter 1), this translator gets lost. When it tries to write Chapter 9, it can't "see" Chapter 8 because, in its logic, Chapter 8 hasn't been written yet.
The Result: When the AI tries to generate video backward, the scenes "glitch" or flicker at the boundaries where one block of video meets the next, because the translator is missing the context it needs.

The Solution: The "Ghost Pages" (Blockwise Anchor Latents)

To fix this glitch without rebuilding the entire translator (which would be too hard and slow), the researchers invented a clever trick called Blockwise Anchor Latents.

The Analogy: Imagine you are writing the backward story. Even though you haven't written the previous chapter yet, you tape a few "ghost pages" to the left side of your current page. These ghost pages aren't part of the final story you show the audience; they are just placeholders that remind the translator what the previous scene looked like.
How it works: The AI generates a small block of video (the real scene) along with these extra "anchor" latents (the ghost pages). The AI uses the anchors to understand the context, writes the real scene smoothly, and then throws the anchors away.
The Result: The video flows perfectly backward without any flickering or jumps, even though the underlying translator still thinks it's only supposed to look forward.

The "Swiss Army Knife" Model

Usually, if you want an AI to write forward, you train one model. If you want it to write backward, you train a different one. If you want it to fill in the middle, you train a third.

UniTemp is different. It is a single, unified model that learned to do all three at once.

Forward: It can extend a video into the future.
Backward: It can create a prequel or extend a video into the past.
In-between: It can take two separate clips (like a "Start" and an "End") and invent the missing scenes that connect them.

What Can You Do With This?

The paper shows that with this single model, you can do things that were previously impossible or required multiple different tools:

Looping Videos: You can take the end of a video and seamlessly connect it back to the beginning to make an endless loop.
Scene Transitions: You can take two very different scenes (e.g., a forest and a city) and have the AI invent a smooth transition between them.
Visual Stories: Instead of just watching a story unfold, you can pick key scenes (Scene 1, Scene 3, Scene 5) and ask the AI to fill in the gaps (Scene 2 and Scene 4) or write the ending, all in any order you choose.

Summary

UniTemp is like giving a video writer a "rewind" button and a "fill-in-the-blanks" tool. It solves the technical glitch of backward generation by using temporary "ghost pages" to keep the story smooth, resulting in one smart model that can handle video creation in any direction you need.

Technical Summary: UniTemp

Problem Statement

Autoregressive video diffusion models have emerged as a promising approach for efficient, long-video generation by utilizing block-wise generation and key-value (KV) caching. However, existing methods are restricted to forward (causal) temporal generation, where each block is conditioned exclusively on previously generated content. This limitation hinders practical video creation workflows that require flexible temporal ordering, such as:

Backward extension: Generating a prologue conditioned on an existing clip.
Inbetween generation: Filling temporal gaps between two segments.
Visual storytelling: Generating key frames first and connecting them.

Attempts to naively extend forward-only models to support backward generation result in severe inter-block flickering and discontinuities, particularly in high-motion scenes. The paper identifies the root cause of this artifact as the Causal 3D VAE architecture widely used in state-of-the-art video diffusion models (e.g., CogVideoX, HunyuanVideo, Wan). The Causal 3D VAE encodes latents strictly conditioned on past context. While this aligns with forward generation, it creates a fundamental mismatch during backward generation: the decoder expects past context that is unavailable, leading to visible boundary artifacts. Retraining a non-causal VAE is impractical as it would break compatibility with existing pre-trained teacher models.

Methodology: UniTemp

The authors propose UniTemp, a unified bidirectional distillation framework that trains a single autoregressive student model capable of generating video in any temporal direction (forward, backward, or inbetween) without modifying the pre-trained teacher or VAE.

1. Blockwise Anchor Latents

To address the inter-block flickering caused by the causal VAE's lack of past context during backward generation, the authors introduce blockwise anchor latents.

Mechanism: During backward generation, the block size is expanded from $B$ to $(P + B)$ , where $P$ is the number of anchor latents. These anchor latents represent preceding video tokens.
Function: The expanded block is jointly denoised using full-sequence self-attention. This allows the target block to attend to a proxy of the missing past context (the anchor latents) even when generating in reverse order.
Discarding: After denoising, the anchor latents are discarded and not included in the final output. They serve solely to stabilize the generation of the target block by providing the necessary context for the causal decoder.
Result: This mechanism effectively restores the missing past context at block boundaries, substantially reducing visual artifacts and inter-block flickering.

2. Unified Bidirectional Distillation

UniTemp employs an asymmetric distillation framework (building on Self-Forcing) to train a single student model ( $G_\theta$ ) supervised by a frozen teacher diffusion model.

Training: The student is trained on both forward and backward generation tasks using a shared model and a shared "fake" critic ( $\mu_{fake}^\phi$ $μ_{f ak e}^{ϕ}$ ).
- Forward: Follows standard block-wise autoregressive rollout.
- Backward: Uses the blockwise anchor latents to expand the context window.
Loss: The framework utilizes Distribution Matching Distillation (DMD) to match the student's output distribution to the teacher's, updating the student via a gradient derived from the difference between the teacher's score and the fake critic's score.
Efficiency: By sharing models across directions, the framework avoids the memory costs of separate generators and reduces the performance gap between forward and backward generation.

3. Versatile Test-Time Conditioning

The unified architecture enables flexible conditioning at inference time:

Any-Direction Generation: The model can condition on arbitrary past, future, or both past and future KV caches.
Both-End-Guided Sink Latents: The framework allows placing "sink latents" (stabilizing tokens) at both the beginning and end of a sequence to control content variation over long durations.
Inbetween Generation: The model can infill content between a head block and a tail block by conditioning on their respective KV caches, regardless of the block size used during training (provided the attention horizon is within training limits).

Key Contributions

UniTemp Framework: A unified distillation framework enabling autoregressive video generation in any temporal direction (forward, backward, inbetween) using a single model.
Blockwise Anchor Latents: A novel mechanism to mitigate inter-block flickering in backward generation caused by Causal 3D VAEs, restoring missing past context without retraining the VAE.
Unified Training Strategy: A shared-model approach that maintains training efficiency and supports flexible test-time conditioning, outperforming separate-model baselines.
Demonstrated Capabilities: The framework unlocks diverse workflows including bidirectional video extension, scene transitions, looping video generation, and visual story generation, all without task-specific fine-tuning.

Results

Experiments demonstrate that UniTemp achieves competitive performance compared to forward-only methods while enabling new capabilities:

Short Video Generation: On 5-second videos, UniTemp achieves performance comparable to the Self-Forcing baseline in both forward and backward directions, with slightly higher scores in backward generation.
Long Video Generation: UniTemp supports stable long-video generation (up to 100s). By utilizing both-end-guided sink latents, it effectively controls content drift and maintains high subject consistency and aesthetic quality, outperforming training-based baselines in dynamic content generation.
Inbetween Generation: UniTemp outperforms specialized models like Wan FLF2V and Generative Inbetweening (GI) in background consistency, image quality, and video distribution. It achieves superior boundary consistency by reusing conditioning frame latents rather than regenerating them.
Artifact Reduction: The introduction of anchor latents reduces the inter-block flickering ratio in backward generation from 1.42 (baseline) to 1.07 (with $P=3$ ), approaching the ideal value of 1.0.

Significance

The paper claims that UniTemp bridges the gap between the efficiency of autoregressive video generation and the flexibility required for real-world video creation. By solving the specific technical challenge of causal VAE incompatibility with backward generation via anchor latents, UniTemp enables a single model to handle diverse temporal conditioning modes. This eliminates the need for multiple specialized models or computationally expensive full-sequence attention methods, offering a scalable solution for complex video editing and storytelling tasks.

Limitations

The authors acknowledge that backward generation incurs additional per-block computation due to the joint denoising of anchor latents. Furthermore, the causal VAE and teacher model remain inherently forward-biased; while anchor latents mitigate the resulting asymmetry, they do not fully eliminate it. The authors suggest exploring bidirectional VAE architectures compatible with existing pre-trained teachers as a direction for future work.