Technical Summary: AtlasVid

Problem Statement

Recent diffusion-based video generators have achieved high visual fidelity and prompt controllability, yet scaling them to ultra-high-resolution (UHR) and long-duration videos remains prohibitively expensive. The primary bottleneck lies in the quadratic computational complexity ($O((HWT)^2)$) of the full-attention mechanism in Diffusion Transformers (DiT), where $H$, $W$, and $T$ represent height, width, and temporal length. Doubling these dimensions increases computational cost by 64×.

Existing approaches face significant trade-offs:

• Native High-Resolution Training: Requires massive datasets of native 4K+ long videos and extensive compute resources (e.g., 32–64 GPU clusters), which are scarce and expensive.
• Generation-then-Super-Resolution: "Pseudo high-resolution" pipelines improve sharpness but often fail to recover rich high-frequency details or correct semantic errors.
• Autoregressive Stitching: Generating clips sequentially often leads to inconsistencies in global temporal coherence and scene structure.
• Long Single-Shot Generation: Generating continuous, long-duration videos (e.g., >10s) at 4K+ resolution without clip transitions remains largely out of reach for current models.

Methodology: AtlasVid

AtlasVid proposes a decoupled global-local modeling framework that separates the generation of global semantic structure from local high-frequency details. This approach enables resolution-agnostic training, allowing a model trained primarily at 720P to generalize directly to 4K and beyond.

1. Decoupled Two-Stage Pipeline

The framework operates in two distinct stages:

• Stage 1: Global Semantic Proxy Generation.
  • A semantic generator produces a low-resolution, low-frame-rate video that serves as a global semantic proxy.
  • To extend the temporal horizon without increasing token count, the method employs temporally scaled RoPE (Rotary Positional Embeddings). By scaling the temporal indices by a factor $r_t$ (e.g., $r_t=4$), the model interprets adjacent frames as being separated by larger time intervals. This allows a short sequence (e.g., 81 frames) to represent a long duration (e.g., 20 seconds) while maintaining the same computational cost as a shorter video.
• Stage 2: High-Resolution Detail Generation.
  • Guided by the global proxy, a detail branch performs joint denoising to generate the final UHR video.
  • This stage utilizes a hierarchical locality-preserving attention mechanism. Instead of applying full attention or naive window attention on a flattened token sequence, the latent video volume is partitioned into spatiotemporal cubes. Tokens are reordered so that tokens within the same cube are contiguous.
  • Locality-Preserving Attention: Window attention is applied to these reordered sequences, ensuring that attention windows correspond to actual geometric neighborhoods in space and time. This preserves the pretrained model's ability to synthesize fine local details while reducing complexity.

2. Efficient Joint Denoising

• Asymmetric Hierarchical Attention: The global proxy tokens ($X_{global}$) and detail tokens ($X_{detail}$) are concatenated. The attention mask is designed asymmetrically:
  • $X_{global}$ attends only to itself (unrestricted).
  • $X_{detail}$ attends to its own local neighborhood (via locality-preserving attention) and to the aligned global semantic tokens.
  • $X_{global}$ is prevented from attending to $X_{detail}$ (mask value $-\infty$), enforcing a one-way semantic guidance flow.
• Positional Alignment: To ensure the global proxy guides the correct spatial-temporal regions, the RoPE coordinates of the global tokens are scaled by the resolution ratios ($W/w, H/h, T/t$) to map them to the high-resolution latent grid.
• Training Strategy: The model is trained using LoRA (Low-Rank Adaptation) on 720P data. The feed-forward layers in the detail branch are kept frozen to preserve pretrained local generative capabilities, while only the attention projections and global conditioning layers are adapted. An optional, lightweight refinement stage can fine-tune the feed-forward layers using short 4K clips.

3. Inference Efficiency

• Classifier-Free Guidance (CFG) Removal: Because the global proxy provides a deterministic semantic target, AtlasVid removes the need for CFG during inference, significantly speeding up generation.
• Resolution Agnosticism: The model trained at 720P generalizes directly to 4K and 8K without requiring native high-resolution training data for the main generation task.

Key Contributions

1. Decoupled Global-Local Modeling: A novel framework that separates global semantic modeling from local detail synthesis, addressing the quadratic scaling bottleneck of DiT architectures.
2. Resolution-Agnostic Training: The first method (to the authors' knowledge) to jointly scale spatial resolution (up to 4K+) and temporal duration (up to 321 frames) without relying on massive native 4K training datasets. The model is trained at 720P and generalizes to higher resolutions.
3. Efficiency and Accessibility:
   • Achieves 60.9× speedup compared to the Wan2.1-1.3B baseline at 4K.
   • Reduces per-layer attention FLOPs by up to 1208.2× compared to FlashAttention for 8K generation.
   • Requires only 2 NVIDIA RTX Pro 6000 GPUs for training (vs. 32–64 GPUs for other methods) and enables inference on a single GPU within ~29 minutes for 4K 81-frame videos.
4. Locality-Preserving Attention: A mechanism that reorders tokens to maintain geometric locality, enabling effective sparse attention that respects the spatiotemporal structure of the video.

Experimental Results

• Qualitative Performance: AtlasVid generates coherent, long-duration videos (e.g., 8K 29 frames, 4K 161 frames, 2K 321 frames) with stable structures and fine-grained details. It outperforms native 4K generators (like T3-Video and UltraGen) in maintaining temporal consistency and avoiding structural artifacts (e.g., morphing objects, color shifts) over long durations.
• Quantitative Metrics: On VBench and high-resolution metrics (HD-FVD, HD-LPIPS), AtlasVid achieves the best performance among 4K generation methods and remains competitive with 720P base models.
  • HD-FVD: 284.71 (lower is better), outperforming T3-Video (311.29) and UltraGen (432.71).
  • Motion and Semantics: High scores in Subject Consistency, Background Consistency, and Motion Smoothness.
• Ablation Studies:
  • Removing the global proxy leads to inconsistent generation, confirming the necessity of global semantic conditioning.
  • Removing locality-preserving attention causes the model to fail at scaling to 4K, whereas naive block attention cannot generalize.
  • The optional 4K fine-tuning improves naturalness but is not strictly necessary for plausible 4K generation, demonstrating the robustness of the 720P-trained base.

Significance and Claims

The paper claims that AtlasVid fundamentally shifts the paradigm for UHR long-video generation by demonstrating that pretrained models already possess the local visual priors required for high-resolution detail. The primary challenge is not a lack of generative capacity, but the efficiency of modeling long-range dependencies.

By decoupling these tasks, AtlasVid:

• Democratizes Access: Lowers the barrier to entry for UHR video generation, making it feasible for academic groups and independent creators without access to massive GPU clusters.
• Reduces Environmental Impact: Drastically cuts the energy and carbon footprint associated with training and serving UHR models by replacing quadratic attention with linear-complexity locality-preserving variants.
• Enables New Applications: Facilitates the generation of cinematic long takes, documentary-style footage, and professional content that requires both high spatial detail and long temporal coherence without relying on clip stitching.

The authors acknowledge limitations, noting that errors in the Stage 1 semantic proxy can propagate to the final output and that the method is orthogonal to other acceleration techniques like distillation, which could further improve speed.