Technical Summary: DySink – Dynamic Frame Sinks for Autoregressive Long Video Generation

1. Problem Statement

Autoregressive long video generation relies on bounded-memory streaming to manage the quadratic computational cost of bidirectional diffusion transformers. Current state-of-the-art methods (e.g., LongLive, Rolling-Forcing) typically employ a hybrid attention mechanism combining a sliding window for short-term continuity and static frame sinks (early cached frames) as global anchors for long-term consistency.

The paper identifies two critical limitations in this static allocation strategy:

1. Contextual Mismatch: As generation progresses, the visual state often diverges significantly from the early frames. Persistently caching these outdated anchors biases the model toward reproducing initial visual cues rather than adapting to the evolving scene, leading to temporal stagnation.
2. Sink Collapse: In severe cases, the reliance on static sinks triggers a failure mode known as "sink collapse." Cui et al. (2026) attribute this to RoPE-induced phase re-alignment and inter-head attention homogenization, where a large fraction of attention heads simultaneously over-attend to sink frames. This causes generated content to regress toward sink frames, resulting in abrupt scene resets and cyclic motion patterns.

2. Methodology: The DySink Framework

DySink proposes a retrieval-based framework that replaces fixed early-frame anchors with dynamic frame sinks. The system maintains a compact memory bank and selects visually relevant historical frames on-the-fly. The framework consists of three core components:

A. Memory Bank Construction

• Structure: The memory bank stores historical video blocks, where each entry consists of a compact visual descriptor ($f_i$) and the corresponding layer-wise Key-Value (KV) caches ($K_i, V_i$).
• Descriptor Generation: Visual descriptors are created by decoding latent frames to pixel space, encoding them via a frozen visual encoder, and aggregating features via mean pooling and L2 normalization.
• Novelty-Aware Update: To prevent redundancy, a candidate block is admitted only if its similarity to existing entries is below a threshold ($\tau_{dedup}$). If near-duplicates occur, earlier entries are retained as they are less corrupted by error accumulation.

B. Adaptive Retrieval and Injection

• Retrieval Mechanism: For the current generation block, DySink aggregates descriptors from the local sliding window to form a query. It computes cosine similarity against the memory bank to retrieve the top-$k$ most relevant historical blocks.
• Context Injection: The KV caches of the retrieved blocks are concatenated with the local sliding-window KV caches to form the attention context. This allows the model to access long-range structural cues without permanently relying on fixed early frames.

C. Sink Anomaly Gate

To mitigate sink collapse without modifying the RoPE frequency structure (as done in prior work via RoPE jitter), DySink introduces a lightweight per-layer sink anomaly gate:

• Detection Logic: The gate monitors inter-head consensus. It calculates the fraction of attention heads that assign higher affinity to the retrieved context than to the local window.
• Suppression: If the fraction of heads over-attending to the retrieved context exceeds a consensus threshold ($\tau_{gate}$), the gate suppresses the retrieved KV states for that layer, forcing the model to fall back to the local sliding window. This prevents the "homogenization" of attention that leads to collapse.

3. Training Paradigm

DySink follows a two-stage streaming tuning paradigm based on the Wan2.1 architecture:

1. Short-Horizon Distillation: A student model (1.3B) is distilled from a teacher (14B) using Distribution Matching Distillation (DMD) to create a few-step causal generator.
2. Long-Horizon Fine-Tuning: The model is fine-tuned on 60-second streaming rollouts using LoRA. The full DySink pipeline (retrieval + gating) is active during training to ensure consistency between training and inference.

4. Experimental Results

Experiments were conducted on minute-long video generation (50s, 75s, and 100s) using the Wan2.1-T2V-1.3B backbone.

• Dynamic Degree: DySink significantly outperforms strong autoregressive baselines (e.g., Self-Forcing++, LongLive). On 50s videos, it improves Dynamic Degree by 8.16 points over Self-Forcing++ and 21.12 points over LongLive. This trend holds for 75s and 100s generations.
• Temporal Quality: Despite the increased dynamic evolution, DySink achieves higher Temporal Quality scores than baselines, indicating that the dynamic sinks preserve coherence while allowing scene evolution.
• Ablation Studies:
  • Removing the Sink Anomaly Gate leads to a drastic drop in Dynamic Degree (from 60.45 to 37.98) and Temporal Quality, confirming that without the gate, the model reverts to sink-collapse behaviors (cyclic motion, repeated structures).
  • Comparing Static Sinks vs. DySink shows that while static sinks offer some stability, dynamic retrieval provides superior adaptability to evolving visual states.

5. Significance and Claims

The paper claims that DySink addresses the fundamental rigidity of static memory allocation in autoregressive video generation. By shifting from fixed anchors to content-adaptive retrieval, the method reduces conditioning mismatch and mitigates the specific failure mode of sink collapse.

Key claims include:

• Adaptability: Dynamic sinks allow the model to maintain long-range consistency without being biased toward outdated early-frame attributes.
• Collapse Prevention: The Sink Anomaly Gate effectively suppresses collapse-prone attention patterns by detecting excessive inter-head consensus, offering a solution that does not require modifying the underlying RoPE positional encoding.
• Scalability: The approach demonstrates that minute-long videos can be generated with higher dynamic degrees and temporal quality than current state-of-the-art methods, suggesting that dynamic memory selection is a critical component for future long-horizon generation.

The authors note limitations, acknowledging that the current visual-only indexing may miss fine-grained semantic or object-level cues, and that the full potential of dynamic retrieval may be more pronounced in scenarios with complex, dense events or multi-stage transitions beyond the current 100-second evaluation horizon.