💻 computer science

WorldKV: Efficient World Memory with World Retrieval and Compression

WorldKV is a training-free framework that enables consistent, real-time autoregressive video generation by combining selective retrieval of evicted KV-cache chunks with token compression to maintain long-term world memory while doubling throughput compared to full-KV approaches.

Original authors: Jung Yi, Minjae Kim, Paul Hyunbin Cho, Wooseok Jang, Sangdoo Yun, Seungryong Kim

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jung Yi, Minjae Kim, Paul Hyunbin Cho, Wooseok Jang, Sangdoo Yun, Seungryong Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a video game where the world is generated in real-time by a very smart AI. You can walk around, turn corners, and explore. The goal is for the world to feel "real" and consistent: if you walk out of a room and come back five minutes later, the room should look exactly the same, not like a different room or a blurry mess.

The paper WorldKV tackles a specific problem with how these AI models currently work. Here is the simple breakdown using everyday analogies:

The Problem: The "Short Memory" vs. The "Huge Backpack"

Current AI video models have two main ways of handling memory, and both have flaws:

The "Sliding Window" (Short Memory): Imagine the AI is wearing a blindfold that only lets it see the last few seconds of what it just generated. If you walk away and come back, the AI has "forgotten" the room you left. It tries to guess what's there, often hallucinating (making up) new furniture or changing the walls. This is fast, but the world isn't consistent.
The "Full History" (Huge Backpack): To fix the forgetting, the AI could keep every single thing it has ever seen in its memory. This is like carrying a backpack that gets heavier and heavier with every step you take. Eventually, the backpack becomes so heavy (too much data) that the AI slows down to a crawl, or the computer runs out of space entirely. You get a consistent world, but it stops running in real-time.

The Solution: WorldKV

The authors propose WorldKV, a "training-free" framework. This means they didn't have to re-teach the AI how to learn; they just gave it a smarter way to organize its existing notes. They use two main tricks:

1. World Retrieval: The "Smart Librarian"

Instead of throwing away old memories (like the sliding window) or keeping everything in the immediate workspace (like the heavy backpack), WorldKV acts like a smart librarian.

How it works: When the AI generates a scene, it saves a "chunk" of that memory to a shelf (in the computer's memory) and labels it with the camera angle or the action taken (e.g., "Turned Right").
The Magic: When you walk back into that room later, the AI doesn't look at everything in the library. It checks its label: "Oh, the user is turning Right again." It instantly pulls only the specific memory chunk from the shelf that matches that view and puts it back into its active workspace.
The Result: The AI remembers the room perfectly without having to carry the weight of the entire history at once.

2. World Compression: The "Duplicate File Cleaner"

Even with the librarian, the memory shelf can get crowded because video frames are very similar. If you take 100 photos of a wall while walking slowly, 99 of them look almost identical.

How it works: WorldKV looks at these chunks of memory and asks, "Is this new information, or is it just a copy of what I already have?" It uses a mathematical trick to find "anchor" frames (the most important ones) and deletes the redundant, duplicate parts of the other frames.
The Result: It shrinks the size of every memory chunk by about half. This is like zipping up a folder of duplicate photos. Now, the AI can fit twice as much history on the same shelf without running out of space.

The Results: Best of Both Worlds

The paper tested this on two different AI models (one small, one large) and found that WorldKV achieves the "Goldilocks" zone:

Consistency: It remembers the world almost as well as if it had kept the entire history (Full KV), far better than the "sliding window" method which forgets things.
Speed: It runs about 2 times faster than keeping the entire history, maintaining real-time speeds.
No Extra Training: It works with existing models right out of the box, without needing to retrain them.

Summary Analogy

Think of the AI as a tourist taking a long trip.

Sliding Window: The tourist only remembers the last 5 minutes of the trip. They get lost if they try to return to a place they visited an hour ago.
Full KV: The tourist writes down every single detail of every second of the trip in a massive journal. They never get lost, but they are too slow to walk because they are carrying a 500-pound book.
WorldKV: The tourist keeps a small, active notebook for the last 5 minutes. But, they also have a smart filing cabinet where they store compressed summaries of the rest of the trip. When they need to remember a specific spot, they quickly pull the right summary from the cabinet and add it to their notebook. They never get lost, and they can still walk at a normal pace.

Technical Summary: WorldKV

Problem Statement
Autoregressive video diffusion models have enabled real-time, action-conditioned world generation, yet sustaining a persistent world where revisiting a previously seen viewpoint yields consistent content remains an open challenge. While full KV-cache attention preserves this consistency, it incurs linear growth in memory footprint and attention cost with rollout length, breaking real-time constraints and leading to out-of-memory (OOM) failures. Conversely, sliding-window inference restores throughput by discarding old context but sacrifices long-term consistency, causing models to hallucinate new content or drift when revisiting scenes. Although recent observations suggest that KV caches in these models function as an emergent form of world memory, accessing this memory efficiently without the full computational cost of attending to the entire history is unresolved.

Methodology: WorldKV
The authors propose WorldKV, a training-free framework designed to enable efficient long-term memory in autoregressive video world models. It operates on top of existing sliding-window inference architectures and introduces two complementary components: World Retrieval and World Compression.

World Retrieval:
- Mechanism: Instead of discarding evicted KV-cache chunks, WorldKV stores them in GPU or CPU memory. When the model revisits a scene, it selectively retrieves scene-relevant chunks based on camera or action correspondence and inserts them back into the native attention window.
- Integration: Retrieved chunks are inserted natively without re-encoding or architectural changes. The framework is retrieval-algorithm agnostic, supporting camera/action-based similarity or attention-based strategies.
- Attention Window Structure: The active window is partitioned into four regions: sink KV caches (initial frames), retrieved KV caches (from stored history), recent KV caches (immediately preceding frames), and the current denoising chunk.
World Compression:
- Motivation: Storing all evicted caches is memory-intensive. The authors observe that temporally adjacent frames share substantial visual content, producing near-duplicate KV caches.
- Mechanism: Within each stored chunk (e.g., 3 frames), the first frame is designated as an anchor. The system computes Key-Key cosine similarity between the keys of non-anchor frames and the anchor frame.
- Pruning: Tokens with high similarity to the anchor are pruned as redundant, while low-similarity tokens (representing newly revealed regions or dynamic content) are retained.
- Efficiency: This process compresses each chunk to approximately half its original size (e.g., retaining the anchor plus 25% of tokens from the other two frames), allowing roughly 2× more history to fit within a fixed memory budget.

Key Contributions

World Retrieval: A framework that stores and selectively retrieves evicted KV-cache chunks using camera/action or attention-based signals, enabling the model to access long-term context without architectural modifications.
World Compression: A key-similarity-based pruning mechanism that reduces per-chunk storage by ~50%, enabling double the history coverage under the same memory constraints while preserving or improving revisit fidelity.
Training-Free Efficiency: The approach requires no fine-tuning, distillation, or dedicated memory modules, operating entirely at inference time on frozen backbones.

Experimental Results
The framework was evaluated on two autoregressive video world models of different scales: Matrix-Game-2.0 (1.3B parameters, trained on short sequences) and LingBot-World-Fast (14B parameters, trained on long videos).

Throughput vs. Fidelity: WorldKV matches or exceeds the memory fidelity of full-KV attention while maintaining throughput roughly 2× higher. On LingBot-World-Fast, it achieves performance comparable to full-KV (which is the native mode) but at significantly higher FPS. On Matrix-Game-2.0, it outperforms both sliding-window and full-KV baselines; notably, full-KV underperforms here due to compounding errors from attending to out-of-distribution accumulated caches, whereas WorldKV's selective retrieval mitigates this.
Comparison to SOTA: WorldKV outperforms memory-trained baselines (WorldPlay, Yume-1.5) on metrics like LPIPS, PSNR, and FID without any memory-specific training.
Ablation Studies:
- Intra-chunk compression: Moderate compression (retaining anchor + 50% of other tokens) yields optimal performance; retaining only the anchor degrades fidelity, while retaining all tokens offers diminishing returns.
- Inter-chunk coverage: Compressing more chunks to fit a fixed budget (e.g., 6 chunks $\to$ 3 chunks budget) improves fidelity compared to retrieving fewer chunks at full resolution, highlighting the value of broader historical coverage.

Significance and Claims
The paper claims that WorldKV demonstrates that the KV cache itself is a fundamental, emergent form of world memory, even in models not explicitly trained on long sequences. By efficiently managing this existing resource through retrieval and compression, WorldKV achieves a persistent world model that sustains consistent scene revisits while adhering to real-time inference constraints. The authors emphasize that this is achieved without the latency of 3D reconstruction or the training overhead of dedicated memory modules.

Limitations
The authors note that WorldKV is an inference-time management method; thus, visual fidelity is bounded by the generation quality of the underlying pretrained model. In extremely long rollouts, error accumulation may still occur. Additionally, while CPU offloading can further reduce VRAM usage, current host-device transfer latencies prevent real-time generation in that specific configuration, which is left as future work.