🤖 machine learning

MonarchRT: Efficient Attention for Real-Time Video Generation

Monarch-RT introduces a structured attention parameterization using Monarch matrices to overcome the quadratic cost and representational limitations of existing sparse attention methods in real-time video generation, achieving up to 95% sparsity without quality loss and enabling 16 FPS generation on a single RTX 5090 by outperforming state-of-the-art FlashAttention kernels.

Original authors: Krish Agarwal, Zhuoming Chen, Cheng Luo, Yongqi Chen, Haizhong Zheng, Xun Huang, Atri Rudra, Beidi Chen

Published 2026-02-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Krish Agarwal, Zhuoming Chen, Cheng Luo, Yongqi Chen, Haizhong Zheng, Xun Huang, Atri Rudra, Beidi Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to paint a movie scene, frame by frame, in real-time. You have a magical artist (the AI) who looks at the current frame and the previous ones to decide what to draw next.

The problem is that this artist is incredibly thorough but also incredibly slow. To paint the next frame, they look at every single pixel in the current scene and compare it to every single pixel in the past. If you have 1,000 pixels, they make 1,000,000 comparisons. This is called "quadratic cost," and it's like trying to introduce every person in a stadium to every other person before the game starts. It takes too long for real-time video.

To fix this, previous researchers tried to tell the artist: "Hey, you don't need to look at everyone! Just look at the people sitting next to you (local neighbors) or the people you know well (semantic matches)." They called this Sparse Attention.

But here's the catch: In real-time video generation, this "lazy" approach fails. Why? Because video isn't just about neighbors or old friends. It's about rhythm (the beat of the music, the flow of movement) and sudden surprises (a car turning a corner).

If you only look at neighbors, you miss the rhythm.
If you only look at "known" people, you miss the surprise.
The result? The video looks glitchy, blurry, or the characters morph into weird shapes.

Enter: MonarchRT (The Smart Conductor)

The authors of this paper realized that the artist doesn't need to look at everyone individually, nor do they need to ignore most people. They need a structured way to look at the whole crowd at once.

They introduced MonarchRT, which uses a mathematical trick called Monarch Matrices. Here is how it works, using simple analogies:

1. The "Grid" vs. The "Messy Pile" (Alignment)

Imagine the video frames are a giant grid of Lego bricks.

Old methods tried to grab random bricks or just the ones in a small pile. This broke the structure of the Lego castle.
MonarchRT realizes that the Lego bricks are arranged in specific layers (Time, Height, Width). It says, "Let's group the bricks by their natural layers."
The Analogy: Instead of grabbing a random handful of sand, MonarchRT grabs a perfect, pre-sorted layer of sand. This ensures the "rhythm" of the video (the periodic structure) is preserved perfectly, even if we skip looking at 95% of the individual grains.

2. The "Tile" Strategy (Tiled Monarch)

Sometimes, even a perfect layer has a few weird, unique bricks that don't fit the pattern (like a sudden explosion in the movie).

Old methods would either ignore these weird bricks (making the video look fake) or try to look at every brick (making it slow).
MonarchRT uses a Tiled approach. Imagine the video is a giant floor covered in tiles.
- Most tiles are identical (the sky, the grass). MonarchRT handles these with a simple, fast rule.
- But if a tile has a weird object (a car), MonarchRT zooms in on just that specific tile to handle the detail.
The Benefit: It gets the best of both worlds. It's fast because it treats 95% of the video with simple rules, but it's smart enough to zoom in on the important, weird parts.

3. The "Rehearsal" (Finetuning)

Usually, to get this math right, the computer has to do a lot of "rehearsals" (iterations) every time it draws a frame. This takes too much time.

MonarchRT's trick: They taught the artist how to do this math during the training phase. Now, when it's time to generate the video, the artist doesn't need to rehearse. They just do it in one single step.
The Result: It's like a musician who used to need to tune their instrument for 10 minutes before playing. Now, they just pick it up and play perfectly instantly.

The Grand Finale: Real-Time Magic

Because of these three tricks (Smart Grouping, Tiling, and One-Step Execution), MonarchRT achieves something previously thought impossible:

Speed: It runs 1.4 to 11.8 times faster than the current fastest methods (FlashAttention).
Quality: It maintains 95% sparsity (ignoring 95% of the data) without losing any video quality.
Real-Time: On a powerful consumer graphics card (RTX 5090), it can generate video at 16 frames per second. That means you can type a prompt, and the AI will generate a movie scene as fast as you can type, with no lag.

In summary: MonarchRT is like upgrading from a librarian who has to walk down every single aisle to find a book, to a librarian who knows exactly which shelf the book is on, can grab a whole section of books at once, and has memorized the library layout so they never have to stop and think. It makes real-time AI video generation fast, smooth, and high-quality.

1. Problem Statement

Real-time video generation using Diffusion Transformers (DiTs) faces a critical bottleneck: the quadratic computational cost of 3D self-attention. This is particularly acute in real-time regimes characterized by:

Auto-regressive generation: Errors compound over time, requiring high fidelity at every step.
Few-step diffusion: To achieve low latency, the number of denoising steps is drastically reduced (e.g., 4 steps vs. 50), forcing each step to process significantly more information.

Limitations of Existing Methods:
Prior efficient approximations, such as sparse attention (local neighborhoods or top-k tokens) and low-rank attention, fail in this setting because:

Non-Sparsity: Video attention is not reliably sparse. It contains a mix of pronounced periodic structures (driven by spatiotemporal position) and dynamic, sparse semantic correspondences.
Expressiveness Failure: Oracle top-k attention (selecting the best tokens) still fails to capture the necessary information even with 10% density, leading to severe geometric distortions and quality degradation.
Incompatibility: Existing methods often require retaining >40% of the full attention FLOPs to maintain quality, which is insufficient for true real-time generation (e.g., 16 FPS).

2. Methodology: MonarchRT

The authors propose MonarchRT, a structured attention parameterization that leverages Monarch matrices to factorize the 3D attention mechanism. The core insight is that Monarch matrices can represent a broader family of linear operators (including sparse, low-rank, and structured transforms like FFT) more effectively than standard sparse approximations.

The framework addresses three specific technical challenges:

A. Structural Analysis & Block Alignment

The paper models the 3D attention map $A$ as a sum of a positional component (separable product of distance functions in time, height, and width) and a semantic component (sparse, long-range correlations).

Challenge: Standard Monarch parameterization assumes a specific block structure. If the block sizes do not align with the video's spatiotemporal dimensions (e.g., mixing tokens from different frames or spatial locations), the approximation collapses.
Solution: The authors define an Aligned Monarch Parameterization. They enforce that each video dimension ( $f, h, w$ ) must be entirely contained within exactly one block dimension. For example, choosing block sizes $(fh, w)$ ensures that the separable positional structure is exactly preserved.

B. Tiled Monarch Parameterization

Standard Monarch parameterization suffers from a lack of monotonic refinement: increasing computation (by changing block sizes) does not guarantee better accuracy because it forces a trade-off between dimensions.

Solution: The authors introduce Tiled Monarch Parameterization.
- This generalizes Monarch by decomposing each large block into smaller sub-blocks (tiles).
- It allows for controllable refinement: by increasing the number of tiles, the model can capture sparse, irregular semantic interactions within local neighborhoods without violating the rank-1 assumption of the positional component.
- This provides a strict hierarchy where increased parameters directly correlate with improved expressiveness and accuracy.

C. Finetuning and Efficient Implementation

The original MonarchAttention requires iterative refinement (multiple steps) to converge, which is too slow for real-time inference.

Solution:
- Finetuning: The authors finetune the Monarch factors directly on the video generation task. This allows the model to achieve high-quality approximations with only 1 refinement step during inference, matching the quality of 10-step optimization.
- Custom Kernels: They implemented custom Triton kernels for forward and backward passes. These kernels utilize a "mini-sequence" strategy to manage memory (avoiding quadratic HBM usage for non-causal 3D attention) and leverage SRAM for efficient reduction, similar to FlashAttention.

3. Key Contributions

Theoretical Insight: Demonstrated that 3D video attention combines periodic positional structures and sparse semantic patterns, a hybrid nature that sparse-attention approximations fail to capture but Monarch matrices can represent efficiently.
MonarchRT Framework: Introduced a unified framework featuring Aligned Block Structures and Tiled Monarch Parameterization to ensure both exact representation of positional dependencies and flexible capture of semantic correlations.
Efficiency Optimization: Developed a finetuning strategy that reduces inference-time refinement steps from ~10 to 1, and created custom Triton kernels that outperform existing FlashAttention implementations.
Real-Time Achievement: Successfully enabled true real-time video generation (16 FPS) on consumer-grade hardware (RTX 5090) using an auto-regressive model.

4. Experimental Results

The authors evaluated MonarchRT on state-of-the-art models like Self-Forcing (auto-regressive) and Wan 2.1 (bidirectional).

Quality Preservation:
- MonarchRT achieves 95% attention sparsity (5% of full FLOPs) with no loss in visual quality compared to dense attention.
- On VBench benchmarks, MonarchRT (95% sparse) outperforms dense attention and significantly surpasses other sparse baselines (like VSA, SVG, RadialAttention) which fail to maintain fidelity beyond 85% sparsity.
- In 4-step distilled models, MonarchRT maintains high PSNR, SSIM, and LPIPS scores where other methods degrade severely.
Speed and Latency:
- Kernel Speedups: MonarchRT kernels outperform FlashAttention-2, -3, and -4 on Nvidia RTX 5090, H100, and B200 GPUs.
  - Up to 11.8× speedup over FA-2 on RTX 5090 (720p).
  - Up to 5.6× speedup over FA-3 on H100.
- End-to-End Performance:
  - Achieves 16 FPS real-time generation on a single RTX 5090 using Self-Forcing at 480p with 95% sparsity.
  - Outperforms VSA in both latency and quality at higher sparsity levels (95% vs 85%).

5. Significance

MonarchRT represents a pioneering shift in video generation architecture. It moves beyond the limitations of "sparse attention" by utilizing structured matrix factorization (Monarch) that aligns with the physical and semantic structure of video data.

Scalability: It enables high-fidelity video generation on consumer hardware, making real-time interactive video generation (e.g., for gaming or virtual worlds) feasible without requiring massive server clusters.
Generalizability: The approach of using Tiled Monarch parameterization offers a new paradigm for approximating complex attention mechanisms in other domains where data exhibits both structured periodicity and sparse long-range dependencies.