🤖 AI

Fast Autoregressive Video Diffusion and World Models with Temporal Cache Compression and Sparse Attention

This paper introduces FAST-AR, a training-free framework that accelerates autoregressive video diffusion and world models by compressing temporal caches and sparsifying attention through approximate nearest neighbor matching, achieving 5–10x speedups with constant memory usage while preserving visual quality.

Original authors: Dvir Samuel, Issar Tzachor, Matan Levy, Michael Green, Gal Chechik, Rami Ben-Ari

Published 2026-06-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dvir Samuel, Issar Tzachor, Matan Levy, Michael Green, Gal Chechik, Rami Ben-Ari

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to write a very long story, one sentence at a time. Every time you write a new sentence, you have to read every single sentence you've written before to make sure the new one fits perfectly.

In the world of AI video generation, this is exactly what happens. As the AI creates a video frame by frame, it keeps a "memory bank" (called a KV Cache) of everything it has generated so far. To create the next frame, the AI has to look at this entire growing memory bank.

The problem? As the video gets longer, this memory bank gets huge. The AI has to read more and more text to write the next sentence. This makes the process:

Slower and slower (like trying to find a specific word in a library that keeps adding new books every second).
More expensive (it runs out of computer memory, like a backpack that gets too heavy to carry).

The paper introduces a new method called FAST-AR to fix this. Think of it as giving the AI a set of super-smart shortcuts so it can write long stories without getting tired or losing its memory.

Here are the three "magic tricks" FAST-AR uses:

1. The "Duplicate Finder" (TempCache)

The Problem: In a video, many things stay the same for a long time. If a cat is walking in a garden, the background trees and the cat's fur look almost identical in frame 100 and frame 101. The AI was wasting time remembering the exact same thing twice.
The Solution: FAST-AR acts like a smart librarian who notices, "Hey, I already have a perfect copy of this tree in my memory. I don't need to write it down again."
It compresses the memory by merging these "near-duplicate" moments. Instead of remembering every single frame, it remembers the essence of the scene. This keeps the memory size small and constant, no matter how long the video gets.

2. The "Relevant Reader" (AnnCA)

The Problem: Imagine you are writing a story based on a very long prompt (a detailed description). The prompt might say, "A cat walks, a van passes, then a dog appears." When the AI is currently drawing the "cat," it doesn't need to look at the words "van" or "dog" in the prompt. But old AI models read the whole prompt every single time, wasting energy.
The Solution: FAST-AR uses a "fast search" tool (called Approximate Nearest Neighbor) to instantly figure out: "Which words in the prompt actually matter for this specific frame?"
It ignores the irrelevant words. If the cat is on screen, it only pays attention to the word "cat." This saves a massive amount of computing power.

3. The "Focus Filter" (AnnSA)

The Problem: Inside the video itself, the AI looks at every pixel in relation to every other pixel. It's like trying to talk to everyone in a stadium at once, even though you only need to talk to the person standing next to you.
The Solution: FAST-AR groups similar things together. If a pixel is part of a "cat," it only talks to other pixels that are also part of the "cat." It ignores the background or other objects that aren't related. It's like putting people into small, focused conversation circles instead of one giant, noisy crowd.

The Result: A Marathon Runner, Not a Sprinter

The paper shows that with these three tricks, the AI can generate videos 5 to 10 times faster than before.

Old Way: As the video gets longer, the AI gets slower and slower, eventually running out of memory (like a runner getting tired and stopping).
FAST-AR Way: The AI runs at a steady, fast pace forever. The speed and memory usage stay the same whether the video is 10 seconds or 2 minutes long.

In short: FAST-AR teaches the AI how to stop re-reading the same old notes, ignore the words it doesn't need, and only focus on the people it's talking to. This allows it to create long, high-quality videos without getting bogged down by its own memory.

Technical Summary: FAST-AR

Problem Statement

Autoregressive video diffusion models enable streaming generation, facilitating long-form synthesis, video world models, and interactive neural game engines. However, these models face a critical bottleneck during inference: the KV (Key-Value) cache grows linearly with the number of generated frames. This growth leads to two fundamental issues:

Increasing Latency: As the cache expands, the per-step attention cost scales linearly (and cumulative work scales quadratically, $O(T^2)$ ), causing generation speed to degrade progressively over long rollouts.
Memory Explosion: The expanding cache consumes increasing GPU memory, often forcing short context windows that harm long-range temporal consistency.

Existing solutions from NLP (KV compression for LLMs) do not transfer directly to video diffusion, and recent video-specific methods offer only modest speedups, often at the cost of visual quality or by failing to stabilize memory usage over time.

Methodology: FAST-AR

The authors propose FAST-AR (Fast AutoRegressive diffusion), a unified, training-free attention framework designed to exploit redundancy in autoregressive video generation. The framework consists of three core components, all leveraging fast Approximate Nearest Neighbor (ANN) matching to reduce computation and memory without retraining the underlying model.

1. Motivation: Sources of Redundancy

The authors identify three persistent sources of redundancy in autoregressive video diffusion:

Temporal Key Redundancy: Cached keys across frames are often near-duplicates, as semantic content evolves slowly.
Semantic Query/Key Evolution: Queries and keys are largely semantic and evolve slowly, making many attention score computations redundant.
Cross-Attention Sparsity: In long prompts, only a small subset of tokens is relevant for synthesizing a specific frame; attending to all tokens is computationally wasteful.

2. Core Components

A. TempCache (Temporal Cache Compression)

Goal: Compress the KV cache by identifying and merging temporally corresponding tokens.
Mechanism: Instead of computing full attention matrices to find correspondences, the method uses lightweight ANN (Locality-Sensitive Hashing or Quantization) to find the top-1 nearest neighbor key for each current-frame query in previous frames.
Theoretical Basis: Based on Lemma 5.1, the authors prove that merging identical keys incurs no approximation error if the values are aggregated (mean) and the logits are shifted by the group size. In practice, keys are merged based on a similarity threshold.
Benefit: This bounds the growth of the KV cache, ensuring that memory usage and latency remain constant regardless of generation length.

B. AnnCA (Approximate Nearest-Neighbor Cross-Attention)

Goal: Accelerate cross-attention by selecting only frame-relevant prompt tokens.
Mechanism: The method projects both latent queries and prompt keys into a shared embedding space (LSH or quantized). It identifies prompt tokens that share a bucket with at least one current-frame query. Tokens with no matching bucket are pruned for that frame.
Benefit: Dramatically reduces the number of prompt tokens processed per frame without computing dense attention maps.

C. AnnSA (Approximate Nearest-Neighbor Self-Attention)

Goal: Sparsify self-attention by restricting queries to semantically matched keys.
Mechanism: Reusing the semantic buckets discovered during the cross-attention pruning phase, the method restricts each token's query to attend only to keys within the same semantic bucket(s).
Benefit: Enforces semantic locality and reduces the quadratic cost of self-attention to a sparse, block-sparse operation using efficient kernels (e.g., FlashInfer).

Key Results

Experiments were conducted on state-of-the-art autoregressive models (Rolling-Forcing and LongVie2) using the LongVBench and LongVGenBench datasets.

Speedup: FAST-AR achieves 5× to 10.8× end-to-end speedups compared to dense FlashAttention-3 (FA3).
- TempCache alone provides ~3× speedup.
- Combining TempCache with AnnSA and AnnCA yields the maximum gains (up to ×10.8).
Quality Preservation: The method maintains near-identical visual quality to the dense baseline.
- PSNR, SSIM, and LPIPS scores remain comparable to dense attention.
- VBench scores (perceptual quality) are preserved (~84.0 vs. 84.1 for the baseline).
- In contrast, existing sparse attention baselines (SVG, RadialAttn) significantly degrade quality (VBench drops to ~33–61) or fail to maintain throughput.
Stability:
- Throughput: While dense attention and other baselines slow down as the context grows (3000 frames), FAST-AR maintains constant FPS.
- Memory: Peak GPU memory usage remains nearly constant over long rollouts, whereas baselines exhibit linear memory growth.

Significance and Claims

The paper claims FAST-AR is the first training-free application of ANN-based attention specifically for autoregressive diffusion models. Its primary significance lies in:

Scalability: It enables the generation of multi-minute videos (e.g., 3000 frames) on a single GPU with stable latency and memory, a capability previously restricted by the $O(T^2)$ scaling of attention.
Efficiency without Retraining: Unlike methods requiring fine-tuning or complex training recipes, FAST-AR operates as a plug-in inference framework compatible with existing backbones.
Robustness: It successfully addresses the specific redundancy patterns of video diffusion (temporal correspondence and semantic sparsity) which generic NLP compression or offline video acceleration methods fail to handle effectively.

The authors emphasize that their approach unlocks the practical viability of autoregressive video world models and long-form synthesis by removing the computational and memory barriers that currently limit their rollout length.