🤖 machine learning

Memory-V2V: Memory-Augmented Video-to-Video Diffusion for Consistent Multi-Turn Editing

The paper introduces Memory-V2V, a memory-augmented video-to-video diffusion framework that maintains cross-turn consistency in iterative editing workflows by storing and retrieving prior edits as structured constraints, thereby preventing visual drift while preserving quality with modest computational overhead.

Original authors: Dohun Lee, Chun-Hao Paul Huang, Xuelin Chen, Jong Chul Ye, Duygu Ceylan, Hyeonho Jeong

Published 2026-03-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dohun Lee, Chun-Hao Paul Huang, Xuelin Chen, Jong Chul Ye, Duygu Ceylan, Hyeonho Jeong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are an artist trying to edit a movie. You want to change a character's shirt from blue to red, or you want to move the camera to see the scene from a different angle.

The Problem: The "Forgetful" Editor
Current AI video editors are like talented but incredibly forgetful interns.

Turn 1: You ask them to change a blue shirt to red. They do a great job.
Turn 2: You ask them to change the background to a beach. They do it, but in doing so, they accidentally change the red shirt back to blue, or they make the red shirt look slightly different than before.
Turn 3: You ask for a new camera angle. The shirt is now orange, and the beach looks like a desert.

The AI treats every new request as a brand-new, isolated task. It doesn't remember what it did five minutes ago. This causes the video to "drift" and become inconsistent, like a story where the main character keeps changing clothes and hair color every time the camera cuts.

The Solution: Memory-V2V (The "Super-Intern" with a Notebook)
The paper introduces Memory-V2V, a new system that gives the AI a "long-term memory" and a smart filing system. Instead of starting from scratch every time, it keeps a structured record of all previous edits and uses them as a rulebook for the next step.

Here is how it works, broken down into simple analogies:

1. The "Smart Librarian" (Selective Retrieval)

Imagine you have a library with thousands of books (previous video edits). If you ask the librarian for help, a naive approach is to dump every book on the table and try to read them all at once. This is slow and confusing.

Memory-V2V acts like a smart librarian.

For Camera Angles: If you want to see the scene from a new angle, the librarian looks at your request and says, "Ah, you want to see the left side? I'll only pull the books that show the left side of the room." It ignores the books about the right side or the ceiling.
For Text Edits: If you say "make the apple an orange," the librarian looks for previous edits where an apple was changed to an orange, ignoring edits about cars or trees.

This ensures the AI only looks at the relevant history, saving time and keeping the focus sharp.

2. The "Zoom Lens" (Dynamic Tokenization)

Once the librarian pulls the right books, how do you read them?

The Old Way: You try to read every single word of every book with the same intensity. This is exhausting and slow.
Memory-V2V's Way: It uses a magic zoom lens.
- For the most important parts of the history (e.g., the exact shade of red used in the shirt), it zooms in and reads every tiny detail.
- For the less important parts (e.g., the background sky from 10 edits ago), it zooms out and just reads the summary.

This allows the AI to keep the high-quality details where they matter most without getting bogged down by unnecessary information.

3. The "Noise Cancellation" (Adaptive Token Merging)

Even with the zoom lens, sometimes there is too much information. Imagine trying to listen to a conversation in a crowded room.

The Problem: The AI gets overwhelmed by "noise" (redundant visual data).
The Fix: Memory-V2V has a noise-canceling feature. It identifies which parts of the memory are "quiet" (not important to the current task) and compresses them into a single, small note. It only keeps the "loud" (important) voices clear.

This makes the editing process incredibly fast, even when the AI has to remember a long history of changes.

Why This Matters

Before this, editing a video multiple times was like trying to paint a mural where every time you added a new layer, the old paint would smear or change color.

Memory-V2V is like having a muralist who:

Remembers exactly what color they used yesterday.
Only looks at the relevant parts of the painting to decide what to do next.
Works fast enough to keep up with your ideas.

The Result: You can now edit a video over and over again—changing the weather, the camera angle, and the characters' clothes—without the video looking like a glitchy, inconsistent mess. The "red shirt" stays red, the "orange apple" stays orange, and the whole story remains coherent, no matter how many times you hit "undo" or "redo."

1. Problem Statement

Current video-to-video (V2V) diffusion models excel at single-turn editing (e.g., changing an object's appearance or viewpoint in one pass). However, practical editing workflows are inherently iterative (multi-turn), where users progressively refine outputs over multiple interactions.

The core challenge identified is cross-turn consistency. Existing models treat each editing turn as an independent generation task. When edits are applied sequentially:

Drift: Previously generated regions gradually change appearance or geometry.
Inconsistency: In novel view synthesis, regions not visible in the original input become inconsistent across different camera angles. In long video editing, edited objects (e.g., an apple turned into an orange) appear as different shades or shapes in different segments.
Computational Bottleneck: A naive solution to fix this is to condition the model on all previous edits. However, this leads to prohibitive computational costs (linear growth in tokens) and introduces redundant or irrelevant context.

2. Methodology: Memory-V2V

The authors propose Memory-V2V, a framework that treats prior edits not as temporal continuations (like in long video generation) but as a structured external memory acting as constraints for subsequent generations. The framework relies on three core technical components:

A. Task-Aware Selective Retrieval

Instead of conditioning on the entire history, Memory-V2V retrieves only the most relevant prior videos ( $\tilde{\Omega}$ ) from an external cache ( $\Omega$ ) based on the current editing task.

Video Novel View Synthesis: Uses a Video Field-of-View (FOV) retrieval mechanism. It calculates the geometric overlap between the target camera trajectory and cached videos by sampling points on a unit sphere and checking visibility. Relevance is scored based on intersection-over-union (IoU) and containment metrics.
Text-Guided Long Video Editing: Uses semantic retrieval. Since text prompts can be ambiguous, the system retrieves segments based on the visual similarity (using DINO features) between the source video segments of the current turn and previous turns.

B. Relevance-Aware Dynamic Tokenization

To handle the variable number of retrieved videos without exploding the token count, the system employs dynamic tokenization:

Adaptive Resolution: Retrieved videos are processed by different tokenizers with varying spatio-temporal compression factors (e.g., $1\times2\times2$ , $1\times4\times4$ , $1\times8\times8$ ).
Strategy: The most relevant videos (top-k) are processed with high resolution (fine-grained details), while less relevant videos are aggressively compressed. This ensures the model focuses computational budget on the most critical constraints.

C. Adaptive Token Merging

Even with dynamic tokenization, the quadratic complexity of self-attention in Diffusion Transformers (DiT) remains a bottleneck.

Token Responsiveness: The system analyzes attention maps to determine which frames/tokens are "responsive" (i.e., strongly influence the current generation).
Merging vs. Discarding: Instead of discarding low-responsive tokens (which causes artifacts), the model uses a learnable convolutional operator to merge them into compact representations.
Placement: Merging is applied in the middle and late stages of the DiT blocks (e.g., blocks 10 and 20) where token responsiveness stabilizes, ensuring early layers retain full information for feature extraction.

D. Training Strategy for Long Videos

To train for long video editing without requiring scarce long paired datasets:

The authors use a dataset augmentation pipeline. They take short source-target pairs from existing datasets (e.g., Señorita-2M) and use a generative video extension model to extend the target video.
During training, the model learns to use these extended segments as memory, effectively simulating multi-turn editing on long sequences.

3. Key Contributions

Problem Formulation: Defined multi-turn video editing as a distinct problem requiring consistency across independently denoised iterations, rather than a simple temporal extension.
Memory-V2V Framework: Introduced a memory-augmented architecture where prior edits serve as structured constraints via selective retrieval and relevance-aware compression.
Efficient Mechanisms: Developed VideoFOV retrieval for geometric consistency and semantic retrieval for text editing, coupled with dynamic tokenization and adaptive token merging to scale conditioning without linear cost growth.
Dataset Innovation: Created a pipeline to generate long-form training data from short clips, enabling the training of long-video editors without manual long-video annotations.

4. Experimental Results

The authors evaluated Memory-V2V on two tasks: Video Novel View Synthesis and Text-Guided Long Video Editing.

Novel View Synthesis:
- Consistency: Significantly outperformed baselines (ReCamMaster, TrajCrafter) in cross-iteration consistency (measured by MEt3R and pairwise overlap). While baselines showed drift between the 1st and 3rd iterations, Memory-V2V maintained geometric consistency.
- Quality: Maintained high visual quality and camera adherence comparable to single-turn baselines.
Long Video Editing:
- Consistency: Outperformed state-of-the-art methods (LucyEdit, TokenFlow, RAVE) in Subject Consistency and Background Consistency. It successfully maintained the identity of edited objects (e.g., a specific hat or orange) across >200 frames, whereas others failed to maintain coherence.
- Metrics: Achieved the highest scores in VBench metrics (Aesthetic Quality, Imaging Quality, Motion Smoothness) and DINO/CLIP similarity scores.
Efficiency:
- Compared to naive context-window scaling, Memory-V2V reduced FLOPs and latency by >90%.
- Adaptive token merging provided an additional 30% reduction in computational cost.

5. Significance and Impact

Enabling Practical Workflows: This work bridges the gap between theoretical diffusion models and real-world editing tools, which require iterative refinement. It solves the "drift" problem that currently plagues multi-step video editing.
Scalability: By decoupling memory size from computational cost through selective retrieval and token merging, it makes long-horizon, multi-turn editing feasible on standard hardware.
Generalization: The framework is model-agnostic (demonstrated on ReCamMaster and LucyEdit) and applicable to various video editing paradigms, suggesting a new direction for memory-augmented generative AI.

Limitations: The current model inherits limitations from its base single-turn models (e.g., struggles with extreme view changes) and struggles with multi-shot videos containing abrupt scene transitions, as the training data consists of continuous single-shot videos. Future work aims to address multi-shot editing and integrate with autoregressive frameworks.