💻 computer science

MemRoPE: Training-Free Infinite Video Generation via Evolving Memory Tokens

MemRoPE is a training-free framework that enables high-fidelity, infinite video generation by introducing evolving memory tokens for dynamic context compression and online RoPE indexing to resolve positional conflicts, thereby overcoming the identity drift and motion stagnation inherent in existing autoregressive diffusion methods.

Original authors: Youngrae Kim, Qixin Hu, C. -C. Jay Kuo, Peter A. Beerel

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Youngrae Kim, Qixin Hu, C. -C. Jay Kuo, Peter A. Beerel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to tell a story that lasts for an hour. You have a very talented storyteller (the AI), but they have a terrible memory. Every time they tell a new sentence, they forget the one before it, or they start mixing up the characters' names. By the time they reach the end of the story, the hero has turned into a villain, the setting has changed from a forest to a desert, and the plot makes no sense.

This is exactly the problem with current AI video generators. They are great at making short, 5-second clips, but when you ask them to make a 1-hour video, the quality falls apart. The characters drift, the background glitches, and the motion gets stuck.

MemRoPE is a new "training-free" solution (meaning it doesn't need to relearn anything) that fixes this memory problem. Here is how it works, using some simple analogies:

1. The Problem: The "Sliding Window" vs. The "Black Hole"

Most current AI video makers use a Sliding Window. Imagine you are looking through a narrow tube. You can only see the last 10 seconds of the video. As new seconds come in, the oldest ones fall off the back and are thrown into a black hole forever.

The result: The AI forgets who the main character was 20 minutes ago. It also tries to keep a "static anchor" (like a photo of the first frame) to remember the start, but that photo never updates, so it doesn't help with things that change over time.

2. The Solution: The "Smart Librarian" (Memory Tokens)

MemRoPE introduces a Smart Librarian who doesn't throw books away. Instead, the librarian summarizes them.

The Dual-Stream System: The AI maintains two special "memory buckets":
- The Long-Term Bucket (The Encyclopedia): This bucket slowly absorbs information about the entire video so far. It remembers the main character's face, the general style of the room, and the core story. It updates very slowly, so it doesn't get confused by every tiny movement.
- The Short-Term Bucket (The Sticky Note): This bucket holds the recent action. It remembers what happened in the last few seconds so the video doesn't look frozen or jerky.
How it works: As the video plays, the AI takes the oldest frames that are about to be forgotten, compresses them into a summary, and updates these two buckets. It's like taking a photo of the whole movie so far and keeping it in your pocket, rather than trying to remember every single second.

3. The Secret Sauce: The "Unlabeled Box" (Online RoPE Indexing)

This is the technical magic part, but here's the simple version.

In AI, every piece of information has a "label" telling it where it is in time (like "Frame 1," "Frame 2," etc.). Usually, these labels are glued to the information permanently. If you try to mix information from Frame 1 and Frame 100 together, the labels get confused, and the math breaks.

MemRoPE uses a trick called Online RoPE Indexing:

The Analogy: Imagine you have a box of Lego bricks. Usually, the bricks are stamped with "Part of House A" or "Part of House B." If you try to glue them together, they don't fit.
MemRoPE's Move: It takes the bricks (the video data) and puts them in a box without any stamps. It only puts the stamps on after the bricks are mixed together, right before the AI looks at them.
The Result: The AI can now safely mix the "Long-Term Memory" with the "New Frame" without the math getting confused. It allows the AI to look at the whole hour of video as one continuous, coherent story, even though it's only looking at a small chunk at a time.

Why is this a big deal?

No Re-training: You don't need to teach the AI a new way to learn. You just plug this "Smart Librarian" system into existing models, and they instantly get better at long videos.
Unlimited Length: Because the memory buckets stay the same size (they just get smarter summaries), you can generate a video that is 1 hour, 10 hours, or even longer without the computer running out of memory.
Perfect Consistency: In the paper's examples, MemRoPE generated a 1-hour video of a woman walking through Tokyo. The woman's face, clothes, and the neon lights stayed consistent the whole time. Other methods made her face morph into a monster or the lights turn into static.

In a Nutshell

Think of MemRoPE as giving the AI a photographic memory that updates itself. Instead of forgetting the past or freezing the present, it continuously summarizes the history of the video while keeping the recent action sharp. This allows it to tell a coherent, high-quality story that can last for hours, rather than just a few seconds.

1. Problem Statement

Autoregressive video diffusion models enable real-time frame streaming but face critical limitations when generating long-form content (minutes to hours):

Context Loss & Identity Drift: Standard sliding-window KV caches use FIFO (First-In-First-Out) eviction, discarding past context. This leads to progressive degradation in subject identity, visual fidelity, and motion coherence over time.
Static Anchors: Existing methods (e.g., Deep Forcing) attempt to solve this by preserving initial frames as "attention sinks." However, these static anchors cannot reflect the evolving content of a growing video.
Discrete Token Selection: Methods like Participative Compression (PC) select tokens based on cumulative attention scores. This causes the cache to converge to a stagnant set of tokens, resulting in abrupt visual shifts whenever the cache updates.
Positional Extrapolation: Extending generation beyond the training horizon requires handling Rotary Position Embeddings (RoPE). Standard approaches store keys with RoPE already applied. Averaging these keys (for memory compression) is mathematically invalid because RoPE does not distribute over addition, leading to incompatible rotary phases and broken temporal attention.

2. Methodology: MemRoPE

MemRoPE is a training-free framework designed to enable unbounded video generation with a fixed-size KV cache. It introduces two co-designed components:

A. Memory Tokens (Dual-Stream EMA)

Instead of discarding old frames or selecting discrete tokens, MemRoPE continuously compresses the entire generation history into a fixed-size buffer using Exponential Moving Averages (EMA).

Dual Streams: It maintains two parallel memory streams:
- Long-Term Memory ( $\mu_L$ ): A slow-moving average ( $\alpha_L \approx 0.01$ ) that captures the stable, global identity and scene structure.
- Short-Term Memory ( $\mu_S$ ): A faster-moving average ( $\alpha_S \approx 0.1$ ) that tracks recent dynamics and motion.
Mechanism: As frames exit the local sliding window, their spatially pooled keys and values update these two streams. This ensures the cache always contains a smoothly evolving representation of the video history without increasing memory cost.

B. Online RoPE Indexing

To make the EMA aggregation mathematically valid, MemRoPE decouples content from position.

Position-Free Caching: All keys are stored in the cache without RoPE applied (raw keys). This prevents the mixing of incompatible rotary phases during the averaging process.
Dynamic Indexing: At every attention step, block-relative indices are assigned to the entire sequence (Sink + Memory + Local Window) starting from zero. RoPE is applied on-the-fly to these indices before computing attention.
Benefit: This resolves the positional extrapolation problem (keeping indices within the training range) and makes the temporal aggregation of keys well-defined.

C. Three-Tier Cache Structure

The total KV cache at any step $t$ consists of:

Sink ( $K_{sink}$ ): The initial high-quality frame(s) acting as a static anchor.
Memory ( $K_{mem}$ ): The dual EMA streams ( $\mu_L, \mu_S$ ) representing the compressed history.
Local ( $K_{local}$ ): The recent sliding window of frames.

3. Key Contributions

Training-Free Infinite Generation: MemRoPE enables the generation of videos lasting hours (demonstrated up to 1 hour) using a fixed-size cache, requiring no additional model training.
Memory Tokens: A novel dual-stream EMA mechanism that continuously compresses history into long-term and short-term representations, preserving both global identity and local dynamics.
Online RoPE Indexing: A technique that stores keys without positional encoding and applies RoPE dynamically at inference time. This solves the mathematical incompatibility of averaging RoPE-encoded keys and enables stable temporal aggregation.
Superior Performance: The framework outperforms state-of-the-art training-free methods (Deep Forcing, $\infty$ -RoPE) in visual fidelity, subject consistency, and temporal coherence across minute-to-hour scales.

4. Experimental Results

The authors evaluated MemRoPE on base models Self-Forcing and LongLive using the VBench-Long benchmark.

Quantitative Performance:
- 120s & 240s: MemRoPE achieved the highest average scores across all metrics, particularly in Aesthetic Quality, Imaging Quality, and Subject Consistency.
- 480s & 1 Hour: MemRoPE significantly outperformed $\infty$ -RoPE (the only other method capable of hour-long generation). For example, on LongLive at 1 hour, MemRoPE scored 87.99 vs. $\infty$ -RoPE's 87.01, with notable gains in subject consistency and background stability.
- Comparison: While $\infty$ -RoPE excels in short-term motion smoothness, it fails to maintain long-term identity. Deep Forcing suffers from abrupt visual shifts due to discrete token selection.
Qualitative Results:
- Visual comparisons show that baselines suffer from subject identity drift (e.g., changing number of puppies, color shifts, structural collapse) as video length increases.
- MemRoPE maintains consistent subject appearance, background structure, and color tone throughout 1-hour generations.
User Study & VLM Evaluation:
- In a user study (2AFC), participants preferred MemRoPE over all baselines in 93%+ of cases for color and subject consistency.
- VLM-based evaluation (Gemini 3.1-Pro) confirmed MemRoPE has the highest visual stability scores.
Efficiency: MemRoPE adds negligible inference latency. In some configurations (Self-Forcing), it is actually faster than the baseline because its compact three-tier cache reduces the total number of tokens attended to compared to larger sliding windows.

5. Significance

MemRoPE addresses a fundamental bottleneck in autoregressive video generation: the trade-off between memory constraints and long-term coherence.

Paradigm Shift: It moves away from "retaining more frames" (which is memory-prohibitive) to "remembering better" (via intelligent compression).
Practicality: Being training-free, it can be plugged into any existing autoregressive video diffusion model, making it immediately applicable to current state-of-the-art systems.
Future Applications: This technology enables advanced applications such as continuous world simulation, cinematic long takes, and synthetic data generation for autonomous driving, where maintaining persistent identity over extended durations is critical.

In conclusion, MemRoPE demonstrates that by decoupling positional encoding from content storage and utilizing continuous memory aggregation, it is possible to generate high-fidelity, hour-long videos without the degradation typically associated with autoregressive generation.