Technical Summary: Self Gradient Forcing (SGF)

1. Problem Statement: The Historical Context-Gradient Gap

Recent advances in autoregressive video diffusion have shifted toward Self Forcing, a paradigm where models are trained on histories generated by their own autoregressive rollouts rather than ground-truth video contexts. This approach mitigates exposure bias by aligning training distributions with inference conditions. However, the authors identify a critical limitation in current Self Forcing implementations: the historical context-gradient gap.

In standard Self Forcing, while the model learns to read self-generated history (via the Key-Value cache) to predict future frames, the writing of that history into the cache remains unsupervised. Specifically:

• Frozen KV Cache: During the autoregressive rollout, previously generated latents are processed at a clean context timestep ($t_{ctx}=0$) to produce Key-Value (KV) entries. These entries are stored in the cache as a "frozen" state for future blocks.
• Missing Supervision: Future losses (e.g., Distribution Matching Distillation losses at noisy timesteps) can backpropagate through the reading of the cache (how future tokens attend to history) but cannot backpropagate through the writing of the cache (how earlier latents are encoded into KV representations).
• Consequence: As the rollout extends, the parameters responsible for writing clean-timestep KV entries drift because they are updated by losses at noisy timesteps without direct feedback on whether the resulting memory representation is useful for long-horizon generation. This leads to identity drift, layout inconsistencies, and scene breaks in long videos.

A direct solution—keeping the entire serial rollout graph differentiable to allow future losses to backpropagate through the entire history—is computationally infeasible due to memory explosion as the graph grows with rollout length.

2. Methodology: Self Gradient Forcing (SGF)

The authors propose Self Gradient Forcing (SGF), a two-pass training strategy that restores the missing memory-writing supervision signal without requiring full backpropagation through the serial rollout. SGF is orthogonal to existing forcing improvements and can be applied on top of them.

The Two-Pass Strategy

SGF decomposes the training step into two distinct phases:

Pass 1: No-Gradient Autoregressive Rollout

• The model performs a standard serial autoregressive rollout identical to inference, but with gradients disabled.
• For each block $i$, the model denoises latent $z_i$ using the current historical KV cache.
• At a sampled "exit step" $s$ (a specific denoising timestep), the model records:
  1. The noisy input latents ($Z^*$).
  2. The predicted clean latents ($\tilde{X}_{ctx}$) generated at the exit step.
• The clean latents are processed at $t_{ctx}=0$ to update the serial KV cache for the next block, but this update is detached from the computation graph.

Pass 2: Parallel Context-Gradient Reconstruction

• The model discards the serial cache and reconstructs the computation for the sampled exit step in parallel.
• The recorded clean latents ($\tilde{X}_{ctx}$) from Pass 1 are re-fed into the model as stop-gradient inputs.
• The model re-computes the clean-context hidden states, KV representations, and future-to-context causal attention.
• Crucially, the KV representations generated in this pass are differentiable.
• The model then processes the recorded noisy latents ($Z^*$) to generate predictions, and the resulting loss (e.g., DMD loss) is backpropagated.
• Result: The gradient flows from the future loss, through the target-side denoising, through the attention mechanism, and back into the KV writing computation (the clean-context forward pass). This allows future losses to supervise how self-generated history is encoded into memory.

Technical Implementation Details

• Gradient Boundary: The method avoids full rollout backpropagation (BPTT). Gradients flow only through the bounded Pass-2 reconstruction. The inputs to the reconstruction (the recorded latents) remain stop-gradient, ensuring the optimization focuses on the writing mechanism rather than the rollout trajectory itself.
• Efficiency: By using parallel reconstruction with a static causal mask (e.g., via FlexAttention), SGF avoids the memory blow-up associated with a fully differentiable serial cache.
• Streaming Policy: For frame-wise generation, SGF utilizes a sink-plus-FIFO context policy, maintaining a fixed sink prefix and a window of recent latents to isolate the effect of the gradient forcing.

3. Key Contributions

1. Identification of the Gap: The paper formally identifies the "historical context-gradient gap" in frozen-cache Self Forcing, where future losses supervise cache reading but not the clean-context writing computation, leading to drift in long-horizon generation.
2. SGF Algorithm: Introduction of a two-pass training strategy that recovers memory-writing supervision via parallel context-gradient reconstruction, enabling future losses to train the model to encode context into more effective causal memory.
3. Empirical Validation: Extensive experiments demonstrating that SGF significantly improves native long-video extrapolation (60s and 240s) while maintaining short-horizon (5s) quality comparable to Self Forcing.

4. Experimental Results

The authors evaluated SGF against matched Self Forcing baselines across frame-wise and chunk-wise generation, using various initializations (Causal ODE, Causal CD, Teacher Forcing) and horizons (5s, 60s, 240s).

• Quantitative Metrics:
  • SGF consistently outperforms Self Forcing in subject consistency, background/layout consistency, temporal stability (flickering), and aesthetic quality at 60s and 240s horizons.
  • While Self Forcing sometimes scores higher on "dynamic degree," the authors attribute this to incoherent motion (scene jumps, object deformation) rather than quality. SGF maintains more stable and plausible camera evolution.
  • At the 5s horizon, SGF performs comparably to Self Forcing, confirming that the method does not degrade short-term generation.
• Qualitative Results:
  • Visual comparisons show that Self Forcing models eventually suffer from identity drift, view jumps, and layout breaks as extrapolation continues.
  • SGF models preserve subject identity, pose, and scene layout over 240 seconds, even when trained on only a 5-second window.
• User Study: A blind GSB (Greater-Same-Better) preference study with over 1,900 paired judgments showed a consistent preference for SGF over Self Forcing across all tested settings, with positive preference scores ranging from ~29% to ~48%.
• Training Feasibility: SGF introduces modest overhead. Peak memory increases slightly (from ~79GB to ~87GB), but stable memory decreases (from ~79GB to ~63GB) due to the elimination of the recurrent graph. Runtime increases by approximately 1.3 seconds per 5 training steps.

5. Significance and Claims

The paper claims that SGF addresses a fundamental limitation in current autoregressive video diffusion training: the inability of future losses to supervise the encoding of self-generated history into memory.

• Native Extrapolation: SGF enables models trained on short windows (e.g., 5 seconds) to extrapolate to minute-scale videos (60s, 240s) with significantly improved coherence.
• Orthogonality: The method is presented as a drop-in improvement that can be combined with other forcing techniques (e.g., Causal Forcing, Rolling Forcing) and initialization strategies without conflict.
• Memory-Writing Supervision: The core significance lies in shifting the training objective to ensure that the model learns not just to read its own history, but to write it in a way that remains useful for future generation steps.

The authors conclude that by closing the historical context-gradient gap, SGF provides a scalable path toward high-quality, native long-video generation without the prohibitive memory costs of full differentiable rollouts.