Co-Director: Agentic Generative Video Storytelling

1. Problem Statement

While diffusion models have achieved high-fidelity video generation, transforming these clips into coherent, long-form storytelling remains a significant challenge. Current approaches face three primary limitations:

• Semantic Drift & Cascading Failures: Existing agentic pipelines rely on linear, "waterfall" chains of independent modules (e.g., script $\to$ storyboard $\to$ video). Errors in early stages propagate downstream, causing identity drift (characters changing appearance) and narrative incoherence.
• Lack of Global Optimization: Current systems use static, handcrafted prompts. They lack a mechanism to map high-level creative intent to downstream generation, failing to explore novel narrative strategies or balance exploration with exploitation.
• Evaluation Gaps: Existing benchmarks focus on isolated sub-tasks (e.g., text-to-image) or rely on memorized real-world entities, failing to evaluate true end-to-end storytelling capabilities under strict constraints.

2. Methodology: Co-Director Framework

The authors propose Co-Director, a hierarchical multi-agent framework that formalizes video storytelling as a global optimization problem. The system integrates a Multi-Armed Bandit (MAB) for global strategy selection with local self-refinement loops.

A. Hierarchical Multi-Agent Architecture

The system is orchestrated by a central Orchestrator Agent ($\Phi_{orch}$) that manages the flow and global state:

1. Pre-Production Agent ($\Phi_{pre}$): Transforms user prompts and reference visuals into a structured Storyboard ($S$). It generates:
   • Creative Brief: Contextual enrichment.
   • Storyline ($L$): Scene-by-scene script.
   • Visual Assets ($V$): Generation of missing character/prop references.
   • Storyboard: Detailed scene descriptors including camera kinematics and audio directives.
2. Production Agent ($\Phi_{prod}$): Synthesizes multimodal artifacts from the storyboard:
   • Keyframe Agent ($\Phi_{frame}$): Generates initial frames to establish visual priors and prevent identity drift.
   • Video Agent ($\Phi_{video}$): Conditions video diffusion models on keyframes to generate clips ($C$).
   • Audio Agent ($\Phi_{audio}$): Synthesizes voiceover and music based on global directives.
3. Post-Production Agent ($\Phi_{post}$): Assembles clips and audio into the final video ($F$).

B. Global Optimization via Multi-Armed Bandits (MAB)

Instead of linear prompting, Co-Director treats the creative process as a search over a hierarchical parameter space ($\theta$).

• Action Space: Defined by three orthogonal axes:
  1. Creative Strategy ($\theta_{cs}$): Informational, Transformational, or Comparative.
  2. Narrative Mode ($\theta_{nm}$): Analytical, Vignette, or Narrative Drama.
  3. Aesthetic Archetype ($\theta_{aa}$): Clarity/Energy, Cinematic Premium, Minimalist Focus, or Kinetic Grit.
• Mechanism: The MAB (using UCB1 algorithm) selects a combination of these parameters ($\theta^{(t)}$) for each iteration. The Orchestrator synthesizes these into specific instructions injected into sub-agent prompts, ensuring all agents align with a unified creative intent.
• Reward Signal: An MLLM Judge evaluates the final video and outputs a factored reward signal $R = (r_{cs}, r_{nm}, r_{aa})$. This decouples the evaluation of the creative strategy from the execution quality, allowing the MAB to isolate which specific parameters drove success.
• Warm Start: An LLM initializes the MAB's expected values based on the product domain to bias early exploration toward theoretically sound configurations.

C. Local Optimization via Agentic Self-Refinement

To prevent error propagation, the system employs feedback descent loops:

• Storyline Refinement: An LLM evaluates the narrative for hook quality, cohesion, and product integration, rewriting the prompt if scores fall below a threshold.
• Keyframe Refinement: An MLLM evaluates the sequence of keyframes jointly (not individually) to detect identity drift and environmental inconsistencies, selectively regenerating only flawed frames.

3. Key Contributions

1. Co-Director Framework: The first system to formalize generative storytelling as a global optimization problem using MAB-driven steering, moving beyond linear pipelines to a hierarchical, coherent architecture.
2. Hierarchical Parameterization: A novel method for propagating creative intent across agents by decomposing high-level strategies into specific, actionable directives for sub-agents, preventing semantic collisions.
3. GenAd-Bench: A rigorous new benchmark consisting of 400 scenarios across 200 fictional products. It uses fictional entities to prevent model memorization and enforces strict constraints on visual asset fidelity, demographic alignment, and marketing appeal.
4. Factored Reward Mechanism: A novel evaluation approach that decouples strategic efficacy from execution quality, enabling precise credit assignment in non-differentiable pipelines.

4. Experimental Results

The authors evaluated Co-Director on GenAd-Bench against state-of-the-art baselines, including monolithic models (Veo 3.1, Wan 2.6), commercial platforms (HeyGen, Creatify), and other agentic frameworks (AniMaker, MovieAgent).

• Performance: Co-Director achieved the highest average score (81.4/100), significantly outperforming the next best agentic baseline (MovieAgent at 65.3) and monolithic models (Veo 3.1 at 63.6).
• Key Metrics:
  • Visual Asset Fidelity (VAF): 82.1 (vs. 67.0 for Wan 2.6).
  • Demographic Alignment (DA): 91.4 (vs. 81.3 for MovieAgent).
  • Marketing Appeal (MA): 82.0.
  • Visual Quality (VQ): 70.2.
• Ablation Studies:
  • Removing MAB optimization (Random Search) dropped the average score to 75.7.
  • Removing local self-refinement (Storyline or Keyframe) caused significant drops in narrative coherence and asset fidelity (e.g., VAF dropped by ~9.8 points without keyframe refinement).
  • The "Factored Reward" design proved critical; collapsing it into a single scalar reduced performance by 5.0 points.
• Efficiency: The MAB with Warm Start converged to high-quality results in the first few iterations, whereas Random Search required many more iterations to find comparable configurations.

5. Significance

• Principled Creative Automation: Co-Director demonstrates that generative video can move beyond simple prompt-following to strategic creative reasoning, balancing exploration of new ideas with the exploitation of proven styles.
• Scalability for Independent Creators: By automating the complex coordination of script, visual consistency, and direction, the framework democratizes high-end cinematic production, previously accessible only to large teams.
• Robust Evaluation: The introduction of GenAd-Bench sets a new standard for evaluating end-to-end generative workflows, specifically addressing the "memorization bias" in current benchmarks.
• Generalizability: While demonstrated on advertising, the framework's ability to handle strict constraints and maintain identity suggests applicability to broader cinematic narratives, short films, and personalized content creation.

In summary, Co-Director represents a paradigm shift from linear, error-prone generation pipelines to a globally optimized, self-correcting agentic system, achieving state-of-the-art results in narrative coherence, visual consistency, and strategic alignment.