🤖 AI

VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion

This paper introduces VideoMLA, a novel architecture that applies Multi-Head Latent Attention to minute-scale autoregressive video diffusion, achieving a 92.7% reduction in KV cache memory and a 1.23x throughput improvement while maintaining or surpassing baseline quality, even though the method succeeds despite the pretrained attention not being inherently low-rank.

Original authors: Hidir Yesiltepe, Jiazhen Hu, Tuna Han Salih Meral, Adil Kaan Akan, Kaan Oktay, Hoda Eldardiry, Pinar Yanardag

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hidir Yesiltepe, Jiazhen Hu, Tuna Han Salih Meral, Adil Kaan Akan, Kaan Oktay, Hoda Eldardiry, Pinar Yanardag

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to tell a story that lasts for an hour, but you only have a small notebook to write it down. Every time you add a new sentence, you need to remember everything you wrote before to make sure the story makes sense.

In the world of AI video generation, this "notebook" is called the KV Cache. It's a memory bank where the AI stores information about the video frames it has already created so it can generate the next ones.

The Problem: The Notebook is Too Heavy

Current AI video models are like students trying to write an hour-long story, but their notebooks are so heavy they can barely lift them.

The Old Way: For every single frame of video the AI remembers, it writes down a massive, detailed description for every single "brain cell" (head) in its network.
The Result: As the video gets longer (minutes long), this notebook becomes so huge that it runs out of memory. To fix this, most systems just throw away old pages (a sliding window), but the pages they do keep are still incredibly bulky and slow to read.

The Solution: VideoMLA (The "Summary" Notebook)

The paper introduces VideoMLA, a new way to write this story that shrinks the notebook by 92.7%.

Here is how it works using a simple analogy:

1. The "Shared Summary" (Low-Rank Latent)
Imagine you are describing a scene to a group of 12 friends (the 12 "heads" in the AI).

Before: You wrote a unique, 128-page detailed report for each friend. That's $12 \times 128 = 1,536$ pages of information per frame!
VideoMLA: You realize all your friends are listening to the same story. Instead of writing 12 separate reports, you write one single, condensed summary (the "latent") that captures the core essence of the scene. All 12 friends share this one summary.
The Magic: This summary is much smaller (only 192 pages instead of 1,536). It captures the "what" (the content) without the redundancy.

2. The "Separate Map" (Decoupled 3D-RoPE)
The summary tells you what is happening, but not where or when.

Before: The location and time info was mixed into those huge, bulky reports.
VideoMLA: You take the location and time info (like "it's raining on the left side at 2 PM") and put it on a tiny, separate sticky note. This note is also shared by everyone.
The Result: You now have a tiny summary + a tiny sticky note, instead of 12 giant reports.

The Big Surprise: It Works Even When It "Shouldn't"

Usually, scientists use this "summary" trick (called Multi-Head Latent Attention) because they believe the original information is naturally simple and easy to summarize (like a low-rank math problem).

The Paper's Discovery:
The authors checked the original AI model and found something surprising: The original information is NOT simple. It's actually incredibly complex and messy (high "rank").

The Puzzle: If you try to summarize a complex, messy painting with a simple sketch, you usually lose a lot of detail.
The Reality: VideoMLA works anyway! Even though the original data is messy, the AI learns to fit the entire story into the small summary space. It turns out the limit isn't how messy the story is, but how big the notebook is. The AI just adapts to fit the whole story into the small space perfectly.

Why This Matters

By shrinking the notebook:

Longer Videos: The AI can now generate minute-long videos without running out of memory.
Faster Speed: Reading a tiny summary is much faster than reading 12 giant reports. The paper shows this makes the AI 1.23 times faster.
Better Quality: Despite the massive compression, the video quality remains high. The AI doesn't get "static" or blurry; the motion stays smooth and the characters stay consistent.

In a Nutshell

VideoMLA is like realizing you don't need to carry a library of books to tell a story. You just need one well-written summary and a small map. This allows the AI to tell longer, smoother stories much faster, without needing a super-computer to hold the memory.

Technical Summary: VideoMLA

Problem Statement
Long-rollout causal video diffusion models have become the dominant approach for streaming, minute-scale video generation. These models typically rely on a fixed-size sliding-window Key-Value (KV) cache to maintain autoregressive generation. While recent works have innovated within this layout by altering token selection, positional encoding (e.g., Infinity-RoPE), or training strategies (e.g., Self-Forcing), they largely preserve the per-head KV layout. In standard dense attention, every cached token stores full keys and values for all attention heads. For a model like Wan-1.3B, this results in a massive memory footprint (approx. 6.0GB for a 21-latent-frame cache), which limits batch sizes, throughput, and the feasible length of video rollouts. Existing efficient methods either change the attention mechanism entirely (e.g., linear attention) or cache only specific layers, but few address the memory cost of the per-token, per-head KV state across all cached layers.

Methodology: VideoMLA
The paper introduces VideoMLA, the first application of Multi-Head Latent Attention (MLA) to autoregressive video diffusion. VideoMLA fundamentally restructures the KV cache by replacing dense per-head keys and values with a shared low-rank representation.

Shared Low-Rank Content Latent: Instead of storing separate keys ( $K$ ) and values ( $V$ ) for each of the $n_h$ heads, VideoMLA compresses the content of a token $x_t$ into a single shared latent vector $c^{KV}_t$ via a down-projection matrix $W^{KV}_\downarrow$ . This latent has dimension $d_c \ll n_h d_h$ .
On-Demand Reconstruction: During attention computation, the per-head content keys ( $k^{nope}_{t,h}$ ) and values ( $v_{t,h}$ ) are reconstructed from the shared latent using head-specific up-projection matrices ( $W^K_\uparrow, W^V_\uparrow$ ).
Decoupled 3D-RoPE: To handle positional information without folding it into the shared latent (which would prevent head-sharing), the method splits the per-head dimension into a content subspace ( $d^{nope}_h$ ) and a positional subspace ( $d^{rope}_h$ ). A single, head-shared positional key $k^R_t$ is computed and rotated using 3D-RoPE. The cache stores the pair $(c^{KV}_t, k^R_t)$ rather than dense per-head states.
Memory Reduction: This design reduces the per-token cached state from $2n_h d_h$ scalars to $d_c + d^{rope}_h$ . In the default configuration ( $d_c=192, d^{rope}_h=32$ ), this yields a 92.7% reduction in KV cache memory per layer.

Key Findings and Spectral Analysis
A central contribution of the paper is resolving a theoretical puzzle regarding why MLA works in video diffusion.

The Puzzle: In language models, MLA is often motivated by the assumption that pretrained $W^K$ and $W^V$ matrices are intrinsically low-rank. However, spectral analysis of the Wan-1.3B video backbone reveals that the joint operator $[W^K; W^V]$ is not low-rank; its 99%-energy effective rank exceeds 1300, far above practical latent dimensions ( $d_c=192$ ).
The Resolution: The authors demonstrate that the effective rank of the learned composed operator in VideoMLA is determined by the architectural bottleneck ( $d_c$ ), not the pretrained spectrum. Both SVD initialization and random initialization saturate the imposed rank budget from the start, and training preserves this budget while adapting within it. Thus, VideoMLA succeeds not by recovering a hidden low-rank structure, but by imposing a rank budget that the model learns to utilize effectively.

Results

Quality: On VBench, VideoMLA matches short-horizon streaming baselines and achieves the best overall score at long horizons (60s) among evaluated methods. It maintains subject identity, scene structure, and motion dynamics, avoiding the static degradation seen in other compression methods.
Efficiency: The method improves inference throughput by 1.23× on a single B200 GPU compared to baselines.
Scalability: Under a fixed memory budget (B200), VideoMLA enables batch sizes 8.0× larger than dense Multi-Head Attention (MHA) at the default $d_c=192$ , and up to 11.4× larger at lower $d_c$ settings, without out-of-memory (OOM) errors.

Significance and Claims
The paper claims that VideoMLA identifies the per-token KV layout as a direct and effective lever for scaling efficient long-horizon video diffusion, orthogonal to existing methods that focus on token selection or positional reparameterization. By reducing the per-token KV memory by 92.7% while preserving visual fidelity and motion consistency, VideoMLA enables minute-scale video generation with significantly lower latency and higher throughput. The work shifts the design question from "what is the intrinsic rank of video attention?" to "what latent budget preserves video quality?", demonstrating that architectural constraints can successfully guide learning even when the pretrained spectrum is high-rank.