🤖 machine learning

Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generation

This paper introduces Sparse Forcing, a trainable native sparse attention mechanism combined with an efficient GPU kernel (PBSA), which leverages the observation that autoregressive diffusion rollouts concentrate on persistent salient blocks to simultaneously improve long-horizon video generation quality and significantly reduce decoding latency and memory usage.

Original authors: Boxun Xu, Yuming Du, Zichang Liu, Siyu Yang, Ziyang Jiang, Siqi Yan, Rajasi Saha, Albert Pumarola, Wenchen Wang, Peng Li

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Boxun Xu, Yuming Du, Zichang Liu, Siyu Yang, Ziyang Jiang, Siqi Yan, Rajasi Saha, Albert Pumarola, Wenchen Wang, Peng Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to write a very long, complex story, like a novel, but you have a terrible short-term memory. Every time you write a new sentence, you have to read the entire book from page one to remember what happened. If the book is 1,000 pages long, this process becomes incredibly slow and exhausting. Eventually, you might forget the main character's name or the setting, and the story starts to make no sense.

This is exactly the problem computers face when generating long videos. Current AI models try to remember every single frame of the video they've created so far to generate the next one. As the video gets longer, the computer gets overwhelmed, slows down, and starts making mistakes (like a character's face changing shape or the background turning into a blur).

The paper "Sparse Forcing" introduces a clever new way to solve this. Here is the simple breakdown:

1. The Problem: The "Hoarding" Computer

Current video AI models are like a person who refuses to throw away anything. They keep every single detail of the past in their head.

The Result: The computer runs out of memory (RAM) and gets very slow.
The Quality Issue: Because the computer is trying to process too much noise, it starts to "drift." A video of a cat might slowly turn into a dog, or the colors might shift from sunny yellow to muddy brown over time.

2. The Insight: "We Remember Moments, Not Days"

The authors noticed something fascinating about how these AI models actually work. Even though the model can look at everything, it naturally focuses on only a few important things:

Persistent Anchors: Just like you remember the main characters and the setting of a story, the AI naturally focuses on a few key "anchor" frames that define the story (e.g., "a cat in a kitchen").
Local Details: For the immediate next step, it only needs to look at the very recent past (the last few seconds) to keep the motion smooth.

The authors realized: Why keep the whole library of history when you only need a few key books and the one you are currently reading?

3. The Solution: "Sparse Forcing"

They created a new system called Sparse Forcing. Think of it as a smart librarian for the AI's memory.

Instead of keeping every single frame, the AI now uses a two-tier memory system:

The "Hall of Fame" (Persistent Memory): The AI identifies the most important "moments" (like the cat's face or the kitchen table) and keeps them in a special, permanent slot. These are the anchors that stop the video from drifting.
The "Current Page" (Local Window): The AI keeps a small, sliding window of the most recent frames to handle fast movements.
The "Trash Bin": Everything else in the middle is discarded. It's not needed for the story to make sense.

4. The Magic Trick: "Native Sparse Attention"

How does the AI know what to keep and what to throw away?
In the past, engineers had to tell the AI exactly what to keep (like a rigid rulebook). Sparse Forcing is different. It teaches the AI to learn what is important on its own.

It's like teaching a student to take notes. Instead of writing down every word the teacher says, the student learns to identify the key concepts and write those down, ignoring the filler words.
Because the AI learned this skill during training, it gets better at focusing on the right things, making the video look more stable and realistic.

5. The Result: Faster and Better

The paper shows that this method is a game-changer:

Speed: Because the computer isn't reading the whole book every time, it generates video 1.1x to 1.2x faster.
Memory: It uses 42% less memory, meaning you can generate much longer videos without crashing the computer.
Quality: The videos stay consistent. If you ask for a 1-minute video of a panda eating bamboo, the panda stays a panda the whole time, and the colors don't fade away.

The Big Picture Analogy

Imagine you are directing a movie.

Old Way: You force the actors to memorize the entire script from the first day of filming every time they shoot a new scene. They get tired, confused, and start forgetting their lines.
Sparse Forcing Way: You give the actors a "Cheat Sheet." It has the main plot points (the Persistent Memory) and the last scene they filmed (the Local Window). They ignore the rest of the script. They stay focused, the movie flows smoothly, and you can film a 1-hour movie in the time it used to take to film a 5-minute one.

In short: Sparse Forcing teaches AI to stop hoarding useless information and start remembering only the "moments" that matter, resulting in longer, smoother, and faster video generation.

1. Problem Statement

The paper addresses two critical bottlenecks in autoregressive (AR) video diffusion models for long-horizon generation:

Compounding Errors & Instability: In AR rollouts, models condition on their own imperfect predictions. Dense attention mechanisms allow early prediction errors to propagate and amplify across the entire sequence, leading to visual drift, color shifts, and geometric distortion over time.
Computational & Memory Scalability: Standard bidirectional or full-sequence attention incurs quadratic complexity ( $O(N^2)$ ) with respect to context length. For long videos (e.g., 1 minute), the Key-Value (KV) cache becomes prohibitively large (e.g., ~45 GB for a 1.3B model), making real-time inference and training infeasible on standard hardware.

Existing efficiency techniques (quantization, distillation) often focus on reducing diffusion steps but do not fundamentally alter the attention topology to control error propagation or memory growth in AR settings.

2. Methodology: Sparse Forcing

The authors propose Sparse Forcing, a training-and-inference paradigm that leverages native trainable sparse attention to simultaneously improve generation quality and reduce latency.

A. Empirical Observations

The method is grounded in two key observations regarding attention patterns in AR video diffusion:

Emergent Persistency: Attention concentrates on a small, persistent subset of historical "visual blocks" (anchors) that carry global semantic information (e.g., subject identity, scene layout). These form an implicit spatiotemporal memory.
Local Block Sparsity: Within the recent context window, attention exhibits a structured, locally diagonal block-sparse pattern, meaning not all recent tokens are equally relevant to the current query.

B. Core Mechanism: Structured Memory Decomposition

Sparse Forcing decomposes the KV cache into two distinct components:

Persistent Memory ( $P_t$ ): A bounded set of "anchor" blocks (including "sink" blocks and dynamically selected historical blocks) that are retained across the entire generation horizon. These provide stable long-range context.
Local Window ( $L_t$ ): A sliding window of recent blocks containing fine-grained details.

C. Persistent Block-Sparse Attention (PBSA)

The attention mechanism combines these two components:

Global Access: Queries attend densely to the Persistent Memory ( $P_t$ ) to ensure global consistency.
Local Sparsity: Queries attend sparsely to the Local Window ( $L_t$ ). A lightweight coarse-routing mechanism (using compressed block representatives) selects the top- $K$ relevant blocks within the local window, ignoring redundant computations.
Dynamic Update: The persistent set is updated via a Top-C retention strategy. As the local window slides, blocks are evicted; a coarse scoring mechanism evaluates their relevance, and the most important ones are promoted to the persistent memory, while less relevant ones are discarded.

D. Training Strategy

To bridge the train-test gap (where the base model was trained with dense attention), the authors employ Distribution Matching Distillation (DMD).

The model is trained with the exact same dynamic sparse cache and adaptive local attention used during inference.
This ensures the model learns to operate within the constraints of the sparse memory, preventing the compounding errors seen when applying sparse attention to a model trained with dense attention.

E. System Implementation: PBSA Kernel

To make this practical, the authors developed Persistent Block-Sparse Attention (PBSA), a custom GPU kernel (built on ThunderKittens).

It supports irregular sparsity patterns and persistent KV carry-over.
It optimizes memory access by blockifying the latent space, enabling coalesced reads and efficient block-level routing.

3. Key Contributions

Identification of Persistent Anchors: The paper empirically demonstrates that AR video diffusion relies on a sparse set of persistent visual anchors for stability, which naive recency-based methods discard.
Sparse Forcing Paradigm: A novel, trainable sparse attention mechanism that compresses, preserves, and updates these anchors while restricting local computation to a dynamically selected neighborhood.
Efficient Kernel (PBSA): A specialized GPU kernel that accelerates sparse attention and memory updates, enabling low-latency, memory-efficient decoding and training.

4. Experimental Results

The method was evaluated on a 1.3B parameter model (based on Wan2.1) for text-to-video generation.

Short-Video (5s):
- Quality: Improved VBench score by +0.26 over Self-Forcing.
- Efficiency: Achieved a 1.11–1.17× decoding speedup and 42% lower peak KV-cache footprint.
Long-Horizon (20s & 1min):
- Quality: Significant gains in visual consistency. For 1-minute videos, VBench scores improved by +2.74 points.
- Stability: Drastically reduced "drift" (color shifts, object deformation) compared to baselines.
- Efficiency: Delivered 1.22× to 1.27× speedups on longer generations.
Kernel Performance:
- The PBSA kernel achieved speedups ranging from 1.16× to 11.11× over FlashAttention-2 (FA2), with higher gains observed at lower sparsity levels (Top-K = 6.25%) and longer sequences.

5. Significance

Scalability: Sparse Forcing enables the generation of minute-long videos with high fidelity on consumer-grade or standard datacenter GPUs by drastically reducing memory requirements.
Quality-Efficiency Coupling: Unlike traditional sparsity methods that trade quality for speed, Sparse Forcing improves generation quality by structurally limiting the propagation of prediction errors, proving that well-designed sparsity is a quality-enhancing mechanism, not just an optimization.
Real-Time Potential: The combination of reduced latency and memory footprint makes real-time, interactive long-form video generation feasible, a critical step toward spatial intelligence and general-purpose AI.

In summary, Sparse Forcing rethinks the attention mechanism in autoregressive video diffusion by mimicking human memory (retaining key moments while focusing on the immediate present), achieving state-of-the-art results in both visual quality and inference efficiency for long-duration video generation.