🤖 AI

EM-Vid: Training-Free Entity-Centric Memory for Efficient and Consistent Multi-Shot Video Generation

The paper proposes EM-Vid, a training-free method that achieves efficient and consistent multi-shot video generation by utilizing an entity-centric memory bank, sparse token conditioning, and a noise-injection mechanism to maintain subject consistency while reducing computational costs and preventing information leakage.

Original authors: Jente Vandersanden, Matheus Gadelha, Chun-Hao P. Huang, Hyeonho Jeong, Yulia Gryaditskaya

Published 2026-05-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jente Vandersanden, Matheus Gadelha, Chun-Hao P. Huang, Hyeonho Jeong, Yulia Gryaditskaya

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are directing a movie with a very talented, but slightly forgetful, AI assistant. Your goal is to create a long story with many different scenes. You want the main character (let's call him "The Fisherman") to look exactly the same in every shot, but you also want the background to change from a pier to a sunset, and you want the Fisherman to wear different clothes in different scenes.

The problem with current AI video tools is that they are like a messy librarian. When you ask for the next scene, they grab the entire previous video frame and shove it into the story. This includes the Fisherman, yes, but also the specific clouds, the trash on the ground, and the exact angle of the sun. Because the AI is looking at the whole messy picture, it gets confused. It might accidentally copy the trash from the first scene into the sunset scene, or it might refuse to change the Fisherman's shirt because it's too attached to the old picture.

EM-Vid is a new way of organizing the AI's memory to fix this. Here is how it works, using simple analogies:

1. The "Entity Bank" (The Organized Filing Cabinet)

Instead of saving whole video frames (which are like saving a whole room with furniture, dust, and light), EM-Vid saves individual pieces of the story in a special filing cabinet called an Entity Bank.

The Old Way: You save a photo of the whole pier.
The EM-Vid Way: You cut out just the Fisherman, just the Pelican, and just the concrete pier, and put them in separate folders labeled [CH_01], [CH_02], and [SC_01].

Now, when you want to make a new scene, you don't show the AI the whole messy pier photo. You just pull out the specific folders you need. If the script says, "The Fisherman walks to the Pelican," the AI only looks at the Fisherman and Pelican folders. It ignores the rest of the pier because it wasn't asked for it. This keeps the story clean and prevents "leakage" (like the trash or wrong clouds appearing where they shouldn't).

2. The "Script with ID Tags" (The Director's Cheat Sheet)

To make this filing system work, the paper introduces a special way of writing the story. Instead of just writing "A man walks," you write:

"[CH_01] walks to [CH_02]..."

Here, [CH_01] is a unique ID tag for the Fisherman. This acts like a cheat sheet for the AI. It tells the AI exactly which folders to pull from the cabinet for that specific moment. This ensures the Fisherman looks like the Fisherman, not like a random stranger, and it stops the AI from getting confused by irrelevant details.

3. The "Noise Injection" (The Eraser and Redraw Tool)

Sometimes, you want the Fisherman to change something small, like swapping his blue shirt for a purple hoodie.

The Problem: If the AI sees a picture of the Fisherman in a blue shirt, it might stubbornly keep painting him in blue, even if you tell it to change.
The EM-Vid Solution: The system uses a "noise injection" trick. Imagine taking the part of the Fisherman's folder that shows his shirt and spraying it with static noise (like TV snow). This "erases" the memory of the blue shirt. Now, when the AI looks at the folder, it sees the Fisherman's face and body clearly, but the shirt area is fuzzy. When you tell it "wear a purple hoodie," the AI is free to paint the purple hoodie because the old blue shirt memory has been scrambled.

4. Why This Matters (The Result)

By using this method, the paper claims they can:

Save Time and Money: Because the AI only looks at the specific "folders" it needs (the Fisherman and the Pelican) instead of the whole messy room, it works much faster and uses less computer power.
Follow Instructions Better: The AI doesn't get distracted by background clutter, so it follows your story script more accurately.
Keep Characters Consistent: The Fisherman looks the same in every shot, but the world around him can change exactly as you describe.

In short, EM-Vid stops the AI from trying to remember the whole movie at once. Instead, it gives the AI a smart, organized list of "who" and "what" is in the current scene, allowing it to build a consistent, long story without getting confused or messy.

Technical Summary: EM-Vid

Problem Statement

Multi-shot video generation aims to produce coherent, long-form visual narratives where recurring entities (characters, objects, scenes) maintain consistent appearances across diverse shots while strictly adhering to shot-specific text prompts. Existing approaches face significant trade-offs:

Joint Generation: Generating all shots simultaneously ensures global consistency but is computationally expensive and difficult to scale.
Autoregressive Generation: Generating shots sequentially improves efficiency but often suffers from "forgetting" earlier content and accumulating errors over long narratives.
Memory-Based Methods: Recent approaches use previously generated frames as memory to guide generation. However, storing holistic full frames entangles persistent entity information with transient scene context. This leads to:
1. Irrelevant Information Leakage: Background artifacts or transient details from memory frames leak into new generations, causing unintended visual elements.
2. High Computational Cost: Self-attention mechanisms operate over all tokens in dense frames, incurring quadratic complexity.
3. Prompt Adherence Issues: Semantically similar prompts can cause models to "copy-paste" stored appearances rather than following new text instructions, particularly for subtle attribute changes.

Methodology: EM-Vid

EM-Vid proposes a training-free, entity-centric memory representation and inference mechanism designed to disentangle persistent entity information from transient context. It operates on top of a pre-trained memory-frame-guided video generation model (specifically a LoRA-fine-tuned variant of Wan2.2-I2V).

1. Structured Story Script Format

To enable efficient retrieval, the method requires a structured input script where:

Persistent subjects and scenes are assigned unique identifiers (e.g., [CH_01], SC_01).
Each shot contains an abstract prompt (referencing identifiers) and a natural prompt (full text description).
The abstract prompt serves as a query to retrieve only the memory entries relevant to the current shot.

2. Entity-Centric Memory Bank

Instead of storing full frames, EM-Vid maintains an entity bank containing sparse latent patches associated with specific entities.

Initialization: For each entity, the system extracts patches from memory frames (or user references) that overlap with the entity's segmented region (using SAM3).
Storage: Each entry stores the VAE latent patch, its spatial coordinates, the source frame index, and appearance descriptors (DINOv2 features and CLIP region-text similarity).
Sparse Conditioning: During inference, the model retrieves only the patches corresponding to the entities mentioned in the current shot's abstract prompt.

3. Inference-Time Sparse Token Conditioning

To integrate sparse patches into a model expecting dense grids, EM-Vid employs a specific pipeline:

Scattering: Retrieved sparse patches are scattered back into their original spatio-temporal locations within a full latent tensor.
Token Pruning: The tensor is patchified into tokens. Only tokens corresponding to entity-containing patches are retained; others are pruned.
Attention Reduction: The transformer backbone (DiT) processes only these relevant tokens, significantly reducing self-attention costs and interference from irrelevant background information.
Reconstruction: Predicted sparse tokens are scattered back into the dense layout and unpatchified to update the full latent state for the next diffusion step.

4. Memory Update Strategy

The entity bank evolves as new shots are generated:

Candidate Selection: Keyframes from new shots are extracted based on aesthetic scores.
Filtering: New candidates are compared against existing entries using DINOv2 similarity. Candidates are accepted if they offer new information (low similarity) or are sufficiently distinct but relevant, while redundant entries are rejected.
Budget Management: A fixed token budget per entity is enforced. Entries are prioritized based on a relevance-to-cost ratio (CLIP similarity divided by token count), ensuring the bank remains compact while preserving semantically relevant variations.

5. Fine-Grained Appearance Control via Noise Injection

To address "copy-paste" behavior where the model ignores subtle prompt changes (e.g., changing a shirt color), EM-Vid introduces a noise injection mechanism:

Boundary Mitigation: Patches at entity boundaries often contain background pixels. Gaussian noise is injected into these regions before encoding to prevent background leakage.
Attribute Modification: If a prompt specifies a change to a specific attribute (e.g., "black shoes" vs. "brown shoes"), patches corresponding to that region are identified. Noise is injected into these specific patches to weaken the model's reliance on the stored appearance, forcing it to follow the text prompt for those regions.

Key Contributions

Entity-Indexed Memory Representation: A novel memory format that stores sparse latent patches for individual entities, disentangling persistent identity from transient context to reduce information leakage and computational cost.
Training-Free Sparse Token Conditioning: A mechanism allowing pre-trained video models to operate on shot-relevant memory tokens only, maintaining compatibility with original architectures while drastically reducing self-attention overhead.
Progressive Entity-Memory Update: A budget-aware update strategy that integrates new entity observations while pruning redundancy, ensuring long-term consistency without unbounded memory growth.
Noise-Based Appearance Control: A strategy using noise injection to suppress irrelevant boundary information and enable fine-grained attribute modifications, preventing the model from defaulting to stored appearances.
New Evaluation Metrics: Introduction of metrics for cross-shot subject consistency (penalizing naive copy-paste behavior) and background scene alignment (measuring alignment between generated backgrounds and scripted scene descriptions).

Results

The authors evaluate EM-Vid on 20 cinematic story scripts (10 shots each) against baselines including Wan2.2-T2V, FramePack, Phantom, and StoryMem.

Prompt Adherence: EM-Vid achieves the highest ViCLIP scores, demonstrating superior alignment with text prompts compared to baselines that suffer from memory leakage.
Subject Consistency: The method maintains strong cross-shot subject consistency (measured by $CSC^*_{DINO/CLIP}$ ), outperforming autoregressive baselines that accumulate errors. It avoids the "copy-paste" bias seen in naive metrics.
Background Consistency: EM-Vid shows improved background scene alignment ($BG A$), successfully changing backgrounds when the script dictates, unlike StoryMem which often carries over background artifacts.
Efficiency: The sparse memory approach reduces memory token usage by up to 88.9% and provides up to a 73% inference speedup compared to dense full-frame memory baselines.
Visual Quality: While vanilla T2V models may score slightly higher on intra-shot aesthetic metrics (due to lack of long-range constraints), EM-Vid remains competitive while offering superior narrative coherence.

Significance and Claims

The paper claims that EM-Vid provides a scalable and effective solution for consistent, long-form video generation without requiring model retraining. By shifting from holistic frame memory to an entity-centric, sparse representation, the method addresses the fundamental entanglement of identity and context that plagues current memory-based approaches.

The authors emphasize that their approach:

Decouples persistent entity information from transient scene context.
Reduces computational complexity by restricting attention to relevant tokens.
Enables fine-grained control over entity attributes through noise injection, a capability often lacking in memory-guided systems.
Improves the balance between maintaining subject identity and adhering to dynamic scene prompts.

The work is presented as a practical inference-time modification that leverages existing pre-trained models, making it accessible and efficient for generating coherent multi-shot narratives.