Technical Summary: Attend Locally, Remember Linearly (ARL2)

1. Problem Statement

Autoregressive (AR) video diffusion has emerged as a leading paradigm for streaming and interactive video generation due to its causal, chunk-wise formulation. However, existing AR video diffusion models rely on softmax self-attention within Diffusion Transformers (DiTs). This reliance introduces a fundamental scalability bottleneck:

• Quadratic Compute & Linear Memory: The standard attention mechanism incurs $O(N^2)$ computational complexity and $O(N)$ memory usage relative to the sequence length $N$ (video frames).
• KV Cache Growth: To maintain temporal consistency, these models accumulate a Key-Value (KV) cache that grows linearly with every generated block. For a 5-second 480p video, the KV cache can exceed 34 GB, surpassing the model parameters themselves.
• Limitations of Existing Remedies: Current solutions like sparse attention, KV cache quantization, or eviction reduce per-step costs or bound memory but fail to simultaneously eliminate linear memory growth and manage streaming context effectively. Sparse attention still relies on a growing cache; eviction irreversibly discards past context.

The core challenge is to convert a pretrained AR video diffusion model into a hybrid architecture that achieves constant memory ($O(1)$) and linear-time scaling without sacrificing the spatial fidelity of intra-frame interactions or the temporal consistency of inter-frame generation.

2. Methodology: ARL2

The authors propose ARL2 (Attend Locally, Remember Linearly), a hybrid attention module that decomposes self-attention into two distinct branches to address the heterogeneous nature of video attention:

2.1 Hybrid Attention Decomposition

The model splits attention into:

1. Intra-Frame Branch (Attend Locally): Preserves the original bidirectional softmax attention within each frame. This branch captures fine-grained spatial correlations and local dependencies, which are critical for visual fidelity. It reuses the frozen pretrained projections to maintain spatial detail without introducing additional latency, as all tokens in a frame are decoded jointly.
2. Inter-Frame Branch (Remember Linearly): Replaces the growing KV cache with a fixed-size recurrent state ($S$). This branch uses Gated Delta Network (GDN) mechanisms to maintain long-range temporal context.
   • Adaptation: Since pretrained $Q, K, V$ projections are optimized for softmax, the model introduces per-head grouped $1\times1$ feature maps ($\phi_q, \phi_k, \phi_v$) to adapt representations to the linear recurrence domain.
   • State Update: The state $S_N$ is updated using a gated rule: $S_N = \alpha_N S_{N-1} + \beta_N K'_N(V'_N - K'_N S_{N-1})$, where $\alpha$ (forget gate) and $\beta$ (learning rate) are learned per-token projections.
   • Output: The inter-frame output is computed as $O_{inter} = Q'_N S_{N-1}$.

2.2 Memory State Management

To prevent the recurrent state from being corrupted by noisy intermediate steps or introducing intra-frame asymmetry, ARL2 employs two specific mechanisms:

• Frame-Level Access: All tokens within a single frame query the same pre-update state ($S_{N-1}$). The state is updated only after the entire frame is processed. This ensures consistent cross-frame context for all tokens in the current frame.
• Clean-Pass Updates: During the multi-step denoising loop, the model queries the state but does not update it. The state is updated only after the final "clean" pass of the frame is produced. This prevents noisy intermediates from contaminating the long-range memory and reduces state writes from one per denoising step to one per frame.

2.3 Training Pipeline: Two-Stage Distillation

Rather than training from scratch, the authors propose an efficient two-stage distillation pipeline to convert a pretrained Causal Forcing model:

• Stage 1 (Per-Layer Distillation & Selection): Each of the 30 attention layers is distilled independently. The authors introduce Attention Recovery Rate (ARR) analysis to classify layers into "Hybrid-Recoverable" (HR) and "Hybrid-Sensitive" (HS).
  • HR Dimensions: Subject consistency, background consistency, motion smoothness, and aesthetic quality (recoverable >85%).
  • HS Dimensions: Imaging quality (sensitive, recovering only ~25%).
  • Layer Selection: A scoring mechanism ($p_\ell$) prioritizes replacing layers that are less sensitive to the HS dimension, ensuring that critical layers for imaging quality retain softmax attention.
• Stage 2 (Joint Distillation): The hybrid model (with selected layers replaced) is jointly fine-tuned end-to-end against the frozen teacher, matching the predicted velocity field to ensure coherent operation across all layers.

3. Key Contributions

1. First Hybrid AR Video Diffusion: This is the first work to convert a pretrained AR video diffusion model into a hybrid linear attention architecture, enabling streaming generation with constant memory without retraining from scratch.
2. Role-Aware Decomposition: The design explicitly respects the heterogeneity of video attention, retaining bidirectional softmax for spatial detail while using linear recurrence for temporal memory.
3. Stable State Management: The introduction of frame-level access and clean-pass updates solves the specific challenges of applying recurrent linear attention to iterative diffusion denoising, preventing context asymmetry and memory corruption.
4. Efficient Distillation Strategy: A sensitivity-guided, two-stage distillation pipeline that minimizes performance degradation while maximizing efficiency gains.

4. Experimental Results

The authors evaluated ARL2 on the VBench benchmark across various video lengths (81 to 921 frames) using a 1.3B parameter model.

• Efficiency:
  • Memory: Achieved up to 54% memory reduction (e.g., 40.5 GB vs. 87.9 GB for 921 frames) and maintained constant memory usage regardless of video length, whereas baselines failed at 1005 frames due to OOM.
  • Speed: Achieved up to 2.26× wall-clock speedup at 921 frames. Speedup scales with video length as the KV-cache overhead becomes dominant.
• Quality:
  • ARL2 (50% replacement): Matched or exceeded the softmax teacher in overall quality, with specific improvements in Temporal Flickering (+0.99) and Motion Smoothness (+1.35).
  • ARL2 (75% replacement): Maintained competitive quality (within ~1% of baseline on Imaging Quality) while achieving maximum efficiency gains.
  • Comparison: ARL2 (50%) outperformed other distilled models and matched the much larger MAGI-1 (4.5B) with 3.5× fewer parameters.
• Ablations:
  • Removing the hybrid branch or using token-level access degraded performance, confirming the necessity of both branches and frame-level state sharing.
  • Headwise gating provided the best trade-off between performance and parameter efficiency.

5. Significance and Claims

The paper claims that ARL2 fundamentally addresses the scalability bottleneck of autoregressive video diffusion. By decoupling spatial and temporal modeling, it enables long-form video synthesis that was previously infeasible due to memory constraints. The authors emphasize that their approach allows for streaming video generation with constant memory, making it suitable for interactive and real-time applications.

The work demonstrates that it is possible to retain the high-quality spatial modeling of softmax attention while replacing the memory-intensive temporal mechanism with a linear, recurrent state. The proposed two-stage distillation proves that such architectural changes can be efficiently transferred to pretrained models with minimal trainable parameters (<2%) and compute cost (~156 H100 GPU-hours), offering a practical path toward scalable video generation.