Narrative Weaver: Towards Controllable Long-Range Visual Consistency with Multi-Modal Conditioning

The paper introduces "Narrative Weaver," a novel framework that achieves controllable, long-range visual consistency in generative AI by integrating multimodal narrative planning with a dynamic memory bank, validated through extensive experiments and a newly released e-commerce advertising dataset.

The Gist

Imagine you are a director trying to make a movie. You have a script, a cast, and a vision. But there's a problem: every time the camera cuts to a new scene, the actors' faces change, their clothes change color, and the background suddenly looks like a different planet. This is the current state of most AI video generators—they are great at making short, beautiful clips, but they lose their mind when asked to tell a long story.

Enter Narrative Weaver. Think of it as a "Super-Director" AI that finally solves this problem. It doesn't just generate images; it weaves a consistent, long-term story where characters and settings stay true to themselves from the first frame to the last.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Drifting" Actor

Current AI models are like actors who forget their lines and costumes after every take. If you ask them to show a woman in a red dress walking through a park, then sitting on a bench, then walking again, the AI might suddenly make her wear a blue shirt, change her hair color, or teleport her to a beach. This is called "visual drift."

2. The Solution: The "Three-Person Crew"

Narrative Weaver acts like a highly organized film crew with three distinct roles working together:

• The Screenwriter (The MLLM): This is a smart language model that acts as the "Director." Before drawing a single picture, it reads your idea and writes a detailed storyboard. It plans the plot: "First, the woman stands. Then she sits. Then she smiles." It keeps the story logic tight.
• The Memory Bank (The Continuity Supervisor): This is the secret sauce. Imagine a sticky note on the director's desk that says, "Remember: The woman has curly brown hair and a red dress." Every time the AI draws a new frame, it checks this "Memory Bank." It looks at the previous pictures and the original reference to ensure the character hasn't accidentally turned into a different person. It prevents the "drift."
• The Artist (The Diffusion Model): This is the actual painter. It takes the Screenwriter's plan and the Continuity Supervisor's notes to create the actual image. Because it has the plan and the memory, it knows exactly what to paint to keep the story consistent.

3. The Training: Learning in Three Steps

You can't just throw a beginner into a Hollywood production. The authors trained Narrative Weaver in three specific stages, like a student actor:

1. Stage 1 (The Scriptwriter): They taught the AI how to write a good story and plan the shots. It learned to say, "Okay, scene 1 is a close-up, scene 2 is a wide shot."
2. Stage 2 (The Translator): They taught the AI how to translate that text plan into visual concepts. It learned to say, "When the script says 'sunset,' I need to paint orange skies."
3. Stage 3 (The Detail Artist): Finally, they taught it the hard part: Consistency. They showed it thousands of examples of how a character should look in different poses so it could learn to keep the face and clothes the same, no matter what the character is doing.

4. The New "Script Library" (The Dataset)

One of the biggest hurdles was that no one had a good library of "long stories" to teach the AI. Most existing data was just short, random clips.
So, the team built a new library called EAVSD (E-commerce Advertising Video Storyboard Dataset).

• The Analogy: Imagine trying to teach a child to write a novel, but you only gave them post-it notes with single sentences. It's impossible. Narrative Weaver needed a whole library of storyboards. They created 330,000 high-quality "storyboards" (sequences of images) specifically for advertising, where a product must look exactly the same in every single shot. This became the textbook for the AI.

5. Why It Matters: From Posters to Movies

Why do we care?

• For E-commerce: Imagine an ad where a model tries on a jacket. The AI can generate 20 different scenes of that model wearing the exact same jacket in a park, a cafe, and a street, without the jacket changing color or the model's face morphing.
• For Filmmaking: It opens the door to AI-assisted storytelling where you can generate entire short films or comic books with a consistent style and characters, rather than just random, disconnected images.

The Bottom Line

Narrative Weaver is the first AI that understands that a story is more than just a collection of pretty pictures. It's a chain of events where the characters must remain the same. By combining a planner (who knows the story), a memory (who remembers the details), and an artist (who draws the pictures), it creates long, coherent visual narratives that actually make sense.

It's the difference between a child drawing random pictures on a page and a professional animator creating a cohesive cartoon.

Technical Summary

Here is a detailed technical summary of the paper "Narrative Weaver: Towards Controllable Long-Range Visual Consistency with Multi-Modal Conditioning."

1. Problem Statement

Current generative AI models (e.g., Sora, Flux, Midjourney) excel at producing high-fidelity short-form visual content but struggle significantly with long-range visual consistency and narrative coherence across extended sequences.

• Visual Drift: Existing models often fail to maintain consistent character identities, backgrounds, and styles over multiple frames or shots.
• Lack of Planning: Most approaches rely on purely textual conditioning or simple frame-to-frame propagation, lacking a high-level "director" to plan complex storylines.
• Data Scarcity: There is a critical lack of datasets that provide multi-modal conditioning (text + image) mapped to coherent, multi-scene storyboards, which is essential for training models to generate consistent long-form content.
• Application Gap: Real-world use cases like filmmaking and e-commerce advertising require strict visual consistency (brand identity) combined with dynamic narrative progression, which current tools cannot reliably achieve.

2. Methodology: Narrative Weaver Framework

Narrative Weaver is a hybrid Autoregressive (AR) + Diffusion framework designed to unify high-level narrative planning with fine-grained visual control.

A. Architecture Overview

The system operates in two main phases:

1. Autoregressive Planning (The "Director"):

   • Utilizes a Multimodal Large Language Model (MLLM) (specifically Qwen2.5-VL-3B).
   • Input: Initial condition images and user instructions.
   • Output:
     • Narrative Plan ($T$): Textual descriptions of future shots.
     • Learnable Queries ($Q$): Compact, high-level representations encoding historical multimodal context.
   • Mechanism: A Dynamic Causal Attention Mask allows learnable queries to access full context (input, past text, past queries) while restricting text tokens to causal attention. Special tokens (<img>, </img>) guide the model to alternate between text planning and visual generation timing.

2. Diffusion Generation (The "Artist"):

   • Utilizes a Diffusion Transformer (DiT) (specifically Flux.1-Dev).
   • Input: The learnable queries ($Q$) from the MLLM, the current conditioning image features ($f_{cond}$), and a Dynamic Memory Bank.
   • Mechanism: The model generates keyframes by decoding the fused conditioning signal.

B. Key Components

• Dynamic Memory Bank: To prevent visual drift over long sequences, the system caches VAE features of previously generated images. It employs a geometric decay strategy (average pooling with a decay factor $\lambda$) to compress historical features. This ensures the total memory length remains bounded (linear complexity) while anchoring new generations to the initial visual conditions and recent history.
• Progressive Multi-Stage Training: To train efficiently with limited data, the authors propose a three-stage strategy:
  1. Stage 1 (Narrative Planning): Train the MLLM to generate storyboards and determine visual output timing (frozen ViT).
  2. Stage 2 (Semantic Coherence): Pre-train learnable queries on large-scale text-image pairs, then fine-tune on high-resolution data to align the MLLM's queries with the diffusion model's semantic space.
  3. Stage 3 (Fine-Grained Alignment): Fine-tune the diffusion model using the full conditioning signal (Queries + Current Image + Memory Bank) to achieve strict inter-frame consistency.

3. Key Contributions

1. Novel Framework: Introduction of Narrative Weaver, the first holistic solution integrating multimodal narrative planning, fine-grained control, and long-range coherence via a hybrid AR-Diffusion architecture.
2. Dynamic Memory Bank: A novel mechanism to encode historical visual features with geometric decay, effectively preventing visual drift while maintaining linear computational complexity.
3. EAVSD Dataset: Construction and release of the E-commerce Advertising Video Storyboard Dataset (EAVSD).
   • Contains ~330K high-quality images.
   • Features triplets of (Product Image, Description, Marketing Goal) mapped to multi-scene storyboards.
   • Addresses the critical gap in datasets for controllable, long-range visual generation.
4. Efficient Training Strategy: A data-efficient, multi-stage progressive training approach that leverages pre-trained models to achieve state-of-the-art performance with limited task-specific data.

4. Experimental Results

The authors evaluated Narrative Weaver across three scenarios: controllable multi-scene generation, autonomous storytelling, and e-commerce advertising.

• Consistency (Q1):
  • Outperformed baselines (StoryDiffusion, Flux.1-Kontext, Qwen-Image-Edit) in Multi-Shot Style Consistency (MSSC) and Multi-Shot Content Consistency (MSCC) on GPT-4o evaluations.
  • Achieved superior DreamSim scores (lower is better) and high CLIP Scores, indicating better alignment with text and reference images without the "copy-paste" artifacts seen in other models.
  • User studies showed a strong preference for Narrative Weaver's outputs over leading methods.
• Autonomous Planning (Q2):
  • On the CoMM benchmark, Narrative Weaver achieved the best overall scores in Style, Entity, Trend, and Narrative Completeness (CPL), demonstrating its ability to autonomously plan coherent visual sequences from text.
• Real-World Application (Q3):
  • Successfully generated coherent storyboards for e-commerce ads, maintaining strict product identity and brand style across diverse scenes (e.g., a model in different environments).
• Efficiency:
  • The architecture reduces computational complexity from quadratic (in vanilla self-attention) to linear with respect to the number of generated frames, making it scalable for long sequences.

5. Significance and Impact

• Bridging the Gap: Narrative Weaver bridges the divide between short-form generative AI and professional production workflows, enabling the creation of long-form visual narratives with strict consistency.
• Scalability: The linear complexity design makes long-range generation computationally feasible, addressing a major bottleneck in current video generation research.
• Data-Driven Advancement: The release of EAVSD provides a crucial resource for the community to further research controllable, consistent visual generation, particularly for commercial applications.
• Future Direction: While currently focused on image sequences, the architecture is designed to be agnostic, offering a clear path toward extending these capabilities to full video generation with temporal consistency.

In summary, Narrative Weaver represents a significant step forward in generative AI by solving the "long-range consistency" problem through a synergistic combination of LLM-based planning, memory-augmented diffusion, and a specialized training regime, validated by a new large-scale dataset.