Long-Horizon Streaming Video Generation via Hybrid Attention with Decoupled Distillation

1. Problem Statement

The paper addresses two critical bottlenecks in Streaming Video Generation (SVG):

• Loss of Long-Range Context: Existing SVG methods typically rely on Sliding Window Attention (SWA) with Key-Value (KV) caching to enforce causality and efficiency. As the window slides, older frames are evicted from the cache. This causes a progressive loss of distant historical context, leading to error accumulation, temporal drift, and degraded coherence in long-form videos.
• Computational Inefficiency: While SWA reduces memory compared to full bidirectional attention, the quadratic complexity within the local window and the need to maintain large windows for context retention still hinder real-time deployment (often limited to ~20 FPS on high-end GPUs).
• Training Instability: Distilling bidirectional models into autoregressive (AR) streaming models often suffers from mode collapse (e.g., frozen motion, uncontrollable camera drift) and distribution shifts when extrapolating to long sequences.

2. Methodology: Hybrid Forcing

The authors propose Hybrid Forcing, a unified framework that combines a novel attention mechanism with a specialized training pipeline.

A. Hybrid Attention Mechanism

Instead of relying solely on a sliding window, the model employs a dual-pathway attention design:

1. Linear Temporal Attention (Long-Range):
   • Function: Preserves global history beyond the sliding window.
   • Mechanism: Instead of storing all evicted tokens, the model maintains a compact, learnable KV state that incrementally aggregates information from evicted frames.
   • Benefit: This allows the model to "remember" the entire video history with negligible memory and computational overhead (constant cost regardless of video length), enabling scalable long-range dependency modeling.
2. Block-Sparse Sliding Window Attention (Local):
   • Function: Refines local temporal interactions within the recent window.
   • Mechanism: It incorporates block-sparse attention into the standard SWA. By calculating block importance scores, the model prunes redundant short-range connections, retaining only the top-K most informative blocks.
   • Benefit: Reduces computational redundancy within the local window, allowing for a smaller window size (e.g., 9 frames) without sacrificing quality, thereby lowering latency.

B. Decoupled Distillation Strategy

To ensure stable optimization of the hybrid architecture, the authors introduce a two-phase distillation process:

1. Phase 1 (Dense Attention): A few-step DMD (Distribution Matching Distillation) phase using dense attention. This allows the model to first learn rich semantic token representations and stabilize autoregressive dynamics without the constraints of sparse or linear attention.
2. Phase 2 (Hybrid Attention): Once semantic representations are consolidated, the model switches to the hybrid attention (linear + block-sparse) for structured historical modeling.
   • Rationale: This separation prevents the premature compression of "noisy" or semantically immature states, which could otherwise trap the model in suboptimal minima.

C. Stability Enhancements

• Relative RoPE Constraint: To prevent distribution shifts when generating videos longer than the training sequence, the authors enforce a consistent upper bound on temporal Rotary Position Embedding (RoPE) indices during both training and inference, representing positions as relative offsets.
• Auxiliary Regularization Loss: To combat mode collapse (e.g., camera drift), an auxiliary loss is introduced. It supervises the student model's first generation step using the teacher model's full rollout from the same initial noise, ensuring the preservation of temporal diversity and motion dynamics.

3. Key Contributions

1. Hybrid Forcing Framework: A novel architecture unifying linear attention for global history retention and block-sparse attention for efficient local modeling.
2. Decoupled Distillation: A training strategy that separates semantic representation learning from structured historical modeling, significantly improving optimization stability.
3. Robust Long-Horizon Training: Solutions for temporal position embedding mismatch and mode collapse via relative RoPE constraints and teacher-guided regularization.
4. Real-Time Performance: The first method to achieve unbounded, real-time video generation (29.5 FPS) on a single NVIDIA H100 GPU without quantization or model compression.

4. Experimental Results

The method was evaluated on short-form (5s) and long-form (30s+) video generation benchmarks (VBench and VBench-Long).

• Performance Metrics:
  • Throughput: Achieved 29.5 FPS on an H100 GPU, significantly outperforming the second-best streaming baseline (21.8 FPS) and bidirectional models (0.78 FPS).
  • Quality: Achieved State-of-the-Art (SOTA) scores across all metrics (Total, Quality, Semantic, Dynamic, Consistency) on both short and long horizons.
  • Long-Form Stability: Unlike baselines that suffer from error accumulation and drift over 30 seconds, Hybrid Forcing maintained high visual fidelity and motion diversity. It successfully generated videos exceeding 10 minutes in length.
• Comparison:
  • Outperformed multi-step autoregressive models (e.g., Pyramid Flow) by a large margin in dynamic metrics (+38.4%).
  • Surpassed the original bidirectional teacher model (Wan2.1) in overall performance while offering ~30x higher throughput.

5. Significance

This work represents a major step forward in making high-fidelity video generation real-time and unbounded.

• Practical Deployment: By achieving 29.5 FPS on a single H100 without quantization, it bridges the gap between research models and practical, interactive applications.
• Architectural Insight: It demonstrates that hybrid attention (combining linear global memory with sparse local processing) is superior to pure sliding window approaches for long-horizon tasks.
• Training Paradigm: The "Decoupled Distillation" strategy offers a new blueprint for training complex AR models, suggesting that separating semantic learning from structural constraints is crucial for stability.

In summary, Hybrid Forcing solves the trade-off between context retention and computational efficiency, enabling the generation of long, coherent, and dynamic videos in real-time.