Technical Summary: AdaState – Self-Evolving Anchors for Streaming Video Generation

1. Problem Statement

Autoregressive video diffusion models generate streaming video by producing frames sequentially, conditioning each new chunk on previously generated content. A fundamental structural limitation in these models is their reliance on a static anchor: the key-value (KV) representation of the first frame occupies a privileged position (position 0) in the attention cache.

Due to the "attention-sink" phenomenon, where causal softmax concentrates mass on initial positions, this static anchor draws disproportionate attention throughout the generation process. While this ensures static consistency, it suppresses video dynamics. The model is effectively locked to the initial viewpoint, preventing natural scene evolution, camera movement, and subject progression. As generation extends, the model struggles to reconcile the evolving scene with the frozen reference, leading to either:

1. Temporal Shallowness: Motion and scene progression are dampened to maintain consistency with the first frame.
2. Drift and Artifacts: If the anchor is removed or updated heuristically, errors accumulate, causing visual collapse or loss of subject identity.

Existing solutions fail to address this root cause. Methods retaining static anchors (e.g., LongLive, Infinity-RoPE) freeze the scene. Methods using content-agnostic updates (e.g., EMA-based approaches like Reward Forcing, MemRoPE) converge to a blurry mean that cannot adapt to specific scene changes. Heuristic replacements (e.g., Rolling Sink) bias generation toward reproducing past frames rather than novel continuations.

2. Methodology: AdaState

The authors propose AdaState, a framework that replaces the static first-frame anchor with an adaptive state—a hidden latent variable that evolves alongside the content.

Core Mechanism

• Adaptive State as a Hidden Latent: Instead of a frozen first frame, the model maintains a state $s$ at position 0 in the KV cache. This state is a hidden latent that is denoised alongside the video content at every chunk but is never rendered to the final video output.
• Joint Denoising and Recurrence: At each generation step $n$, the generator $G_\theta$ jointly denoises the current content frames $x^{(n)}_c$ and the state frames $s_n$ from independent Gaussian noise, conditioned on the KV cache $h_n$.
  • The clean content KV enters the sliding window for short-term context.
  • The clean state KV overwrites position 0 in the cache, serving as the scene reference for the next chunk.
• Relative Time Formulation: Unlike standard generation which encodes absolute time, AdaState treats time as relative. Every generation step sees the same positional structure regardless of the rollout length. The state transition is identical at every chunk, turning the sequential autoregressive process into a recurrent process where:
  • Denoising acts as the transition function.
  • The KV Cache acts as the carrier for the hidden state.
  • No external modules, gating mechanisms, or separate memory units are required.

Horizon-Weighted Training

To ensure the model learns to utilize the adaptive state effectively, the authors introduce Horizon-Weighted Distribution Matching Distillation (DMD).

• Problem: In autoregressive rollouts, errors accumulate in later frames. Standard uniform loss weighting allows the optimizer to focus on clean, early frames, neglecting the critical later frames where the adaptive state must compensate for the absence of original scene content.
• Solution: A linear ramp weighting $w_i$ is applied to the DMD loss, increasing with the frame index $i$. This forces the optimizer to concentrate capacity on the later frames where the state-mediated coherence is most vital.
  $$L = \frac{1}{2} \sum_{i=0}^{N-1} w_i \left\| \hat{x}_i - \left( \hat{x}_i - \frac{S_{fake} - S_{real}}{\gamma} \right)_{sg} \right\|^2, \quad w_i = 1 + \alpha \cdot \frac{i}{N-1}$$

3. Key Contributions

1. Identification of Structural Bias: The paper identifies that the static anchor in autoregressive video diffusion creates a structural attention bias that suppresses dynamics, locking scene composition to the initial viewpoint.
2. Adaptive State Architecture: The authors propose replacing the static anchor with a hidden adaptive state trained via horizon-weighted DMD. This state evolves with the content, allowing the model to generate its own scene reference at each step.
3. Recurrent Formulation without External Modules: The work demonstrates that denoising itself can function as a recurrence. The adaptive state is a hidden variable updated by the model's own iterative refinement and carried via the KV cache, requiring no external memory or gating mechanisms.

4. Experimental Results

The method was evaluated on a model distilled from Wan2.1-T2V-1.3B using Self-Forcing, trained on 21-frame rollouts and tested at 5 seconds (within horizon) and 30 seconds (6× training horizon).

• Qualitative Performance:
  • Static Anchors (e.g., Infinity-RoPE, Rolling Forcing): Preserve identity but freeze the scene, showing no camera movement or scene progression.
  • No Reference (e.g., Self-Forcing, Causal Forcing): Show high dynamics initially but suffer from color drift and visual artifacts as the rollout extends beyond the training horizon.
  • AdaState: Achieves a unique balance, preserving subject identity while enabling natural scene evolution, camera gliding, and continuous motion even at 30 seconds.
• Quantitative Performance:
  • Within Horizon (5s): AdaState achieves the highest total VBench score and VisionReward, with dynamic degree significantly higher than static-anchor methods.
  • Beyond Horizon (30s): AdaState is the only method that does not degrade in dynamic degree. It occupies the "upper-right" region of the consistency-dynamics tradeoff curve, maintaining high consistency while achieving the highest dynamic degree among all baselines.
  • User Study: In a study with 40 raters, AdaState received the highest scores for both "coherent progression" and "prompt adherence," outperforming no-reference, static-reference, and EMA-reference baselines.

5. Significance and Claims

The paper claims that AdaState fundamentally breaks the consistency-dynamics tradeoff that plagues current streaming video generation. By treating the generation process as a recurrence where the model's own denoising updates a hidden scene reference, AdaState enables:

• Richer Motion: Natural camera movement and subject action that evolve over time.
• Long-Horizon Stability: The ability to generate coherent videos far beyond the training horizon without accumulating drift or freezing the scene.
• Architectural Simplicity: The solution requires no architectural changes to the backbone DiT, no auxiliary losses, and no external memory modules, relying solely on the existing KV cache and attention mechanisms.

The authors position this as a step toward adaptive recurrent state in autoregressive generation, suggesting the formulation could apply to other domains where attention concentrates on fixed past positions. However, they modestly note limitations, such as the heuristic setting of the state capacity (currently one frame) and the need for future work on structured external memory for highly complex, multi-actor scenes.