💻 computer science

DirectorBench: Diagnosing Long-Form Video Generation with Personalized Multi-Agent Evaluation

DirectorBench introduces a personalized multi-agent diagnostic benchmark that evaluates long-form video generation across five dimensions and multiple user profiles to localize specific workflow failures and align more closely with human judgment than traditional aggregate scoring methods.

Original authors: Jiamin Chen, Qianben Chen, Jiawen Zhang, Yidi Wu, Yuchen Li, Xiaokun Zhang, Wangchunshu Zhou, Chen Ma

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiamin Chen, Qianben Chen, Jiawen Zhang, Yidi Wu, Yuchen Li, Xiaokun Zhang, Wangchunshu Zhou, Chen Ma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a movie director. In the past, AI video generators were like talented but short-attention-span actors who could only perform a single, perfect 5-second scene. If you asked for a whole movie, the AI would just stitch those scenes together clumsily, resulting in a jumbled mess where the characters changed clothes, the story made no sense, and the audio was out of sync.

DirectorBench is a new "quality control" system designed specifically to grade these long, complex AI movies. Instead of just giving a single grade (like "B-"), it acts like a specialized film critic panel that breaks the movie down into tiny pieces to find exactly where it went wrong.

Here is how it works, using simple analogies:

1. The Problem: The "One-Size-Fits-All" Grade is Broken

Old ways of testing AI videos were like grading a student's essay based only on their handwriting. They looked at if the pictures were clear or if the video didn't freeze. But for a long movie, you also need to check: Did the story make sense? Did the character's voice match their lips? Did the music fit the mood?

Furthermore, everyone has different tastes. A horror fan wants scary sounds; a romance fan wants emotional music. Old tests gave one single score to everyone, ignoring that a video might be perfect for one person but terrible for another.

2. The Solution: The "Personalized Critic Panel"

DirectorBench solves this by using a team of AI agents (specialized robots) that act as different types of critics.

The Script Agent: Reads the story to see if the plot makes sense.
The Visual Agent: Checks if the lighting, camera angles, and character faces stay consistent.
The Audio Agent: Listens to the music and voiceovers.
The Sync Agent: Checks if the lips move with the words and if the sound matches the action.

Instead of just saying "This movie is 7/10," this panel gives a diagnostic report. It might say: "The story is great, the visuals are pretty, but the transition between Scene 2 and Scene 3 is a disaster." This helps the creators know exactly what to fix.

3. The "User Profile" Twist

Imagine you are ordering a custom cake.

Profile A (The Storyteller): Cares mostly about the plot. They don't mind if the frosting is a little messy, as long as the story is good.
Profile B (The Visual Artist): Cares mostly about how the cake looks. They might forgive a boring story if the design is stunning.

DirectorBench lets you plug in these different "User Profiles." It re-runs the evaluation for each profile. A video might get a low score for the Storyteller but a high score for the Visual Artist. This proves that quality depends on who is watching.

4. What They Discovered (The "Bottlenecks")

The researchers tested several different AI movie-making systems using DirectorBench and found some surprising truths:

The "Seam" Problem: The biggest failure point isn't inside a single scene; it's between scenes. The AI is great at making one shot, but terrible at connecting two shots smoothly. It's like a musician who can play a perfect note but can't play a smooth melody. The "transition quality" was the lowest score across all systems.
The "Brain" vs. The "Hands": They tested different "brains" (the AI models that plan the movie) while keeping the "hands" (the tools that actually draw the video) the same. They found that changing the brain mostly changed the story and the coordination of the movie, but didn't fix the basic visual glitches. The "hands" (the video generation tools) were the limiting factor for visual quality, regardless of how smart the planner was.
The "Average" Lie: When they used a generic, "average" score, they missed huge differences. A video that looked "okay" to a neutral observer might be a total failure for a specific user type.

Summary

DirectorBench is a new tool that stops treating AI video generation like a simple math problem with one right answer. Instead, it treats it like filmmaking: it checks the story, the visuals, the sound, and the transitions separately, and it understands that a "good" movie depends on what the viewer actually wants to see. It helps developers stop guessing and start fixing the specific parts of the movie pipeline that are broken.

Technical Summary: DirectorBench

Problem Statement

Long-form video generation is evolving from short, single-scene synthesis toward minute-long, multi-shot narratives requiring cinematic control, audio integration, and cross-modal synchronization. However, existing evaluation benchmarks are insufficient for this new paradigm. Current metrics primarily focus on local visual fidelity, short-horizon temporal consistency, or generic prompt alignment. They fail to diagnose workflow failures in multi-stage agentic systems, overlook audio and cross-modal coordination, and collapse quality into a single aggregate score that ignores user-dependent preferences. Consequently, there is a lack of diagnostic tools capable of localizing bottlenecks (e.g., transition quality vs. single-frame quality) or evaluating how different user profiles perceive the same generated content.

Methodology: DirectorBench

The authors introduce DirectorBench, a personalized multi-agent diagnostic benchmark designed to evaluate long-form video generation through structured metadata, user profiles, and a hierarchical checkpoint system.

1. Benchmark Construction

Metadata Specifications: The benchmark utilizes 80 structured metadata entries across 7 categories (narrative, commercial, educational, sci-fi, cinematic, action, daily-life). Each entry is a JSON document defining objective requirements for three modalities:
- Text: Three-act story arcs, per-shot scripts, dialogue, and narration.
- Visual: Shot descriptions, camera movements, lighting, and consistency constraints (identity, clothing, scene).
- Audio: Dialogue, lip-sync, background music, sound effects, and tone control.
User Profiles: Seven distinct user profiles ( $u_1 \dots u_7$ $u_{1} \dots u_{7}$ ) represent real-world preferences. Each profile defines:
- Priority Weights: Relative importance of four dimensions (Narrative, Visual, Audio, Cross-modal Synchronization). Note: Stability is treated as a system-level metric, not a user preference.
- Hard Constraints: Non-negotiable requirements (e.g., perfect lip-sync) that trigger significant penalties if violated.
- User Taste: Fine-grained aesthetic preferences and expertise levels used to generate personalized prompts.
Personalized Prompts: An LLM generates specific creation instructions ( $p = g(m, u)$ ) by combining metadata and user profiles, ensuring prompts reflect the user's expertise and priority weights while maintaining equivalent information content.

2. Dynamic Checkpoint Rubric

To handle diverse video types, DirectorBench employs a dynamic checkpoint rubric:

Content Profiling: A vision-language model analyzes representative frames and ASR outputs to generate a "Content Profile" with 18 semantic attributes (e.g., presence of characters, dialogue, scene changes).
Checkpoint Activation: A registry of 40 checkpoints organized into 9 sub-metrics across 5 dimensions (Script, Visual, Audio, Cross-Modal, Stability) is maintained. Based on the Content Profile, only relevant checkpoints are activated (e.g., lip-sync checkpoints are skipped for landscape videos without dialogue).
Scoring: Activated checkpoints are scored on a 1–5 ordinal scale, normalized to [0, 1], and aggregated with content-aware weights.

3. Multi-Agent Evaluation Pipeline

The evaluation is implemented as a Directed Acyclic Graph (DAG) using LangGraph, consisting of four phases:

Phase 0 (Preprocessing): Tools extract structured signals (shot boundaries, optical flow, audio separation, ASR, color histograms). All tool traces are recorded for auditability.
Phase 1 (Specialist Evaluation): Four parallel agents (Script, Visual, Audio, Stability) evaluate their respective dimensions using the tool context and content profile.
Phase 2 (Cross-Modal Evaluation): A dedicated agent evaluates text-video consistency and video-audio synchronization, verifying claims against metadata and tool outputs.
Phase 3 (Diagnosis Synthesis): An orchestrator aggregates scores into a structured Diagnosis Report. This includes:
- Per-dimension and profile-aware overall scores.
- A prioritized list of bottlenecks (low-scoring checkpoints).
- Low-confidence flags for human review.
- Actionable recommendations tied to specific failures.

Key Contributions

DirectorBench Framework: A personalized, multi-agent diagnostic benchmark with 80 metadata entries, 7 user profiles, and 40 checkpoint criteria across five dimensions.
Evaluation Pipeline: A dynamic, tool-augmented pipeline that activates checkpoints based on content, uses confidence-aware aggregation, and produces checkpoint-level diagnosis reports rather than single aggregate scores.
Empirical Analysis: A comprehensive evaluation of 4 agentic workflows, 6 base LLMs, and 7 user profiles, revealing specific failure modes and the impact of personalization on quality assessment.

Experimental Results and Observations

The authors evaluated four workflows (MovieAgent, ViMax, Dreamina, Kling), six base LLMs, and seven user profiles.

Workflow Signatures & Bottlenecks:
- Between-Unit Bottlenecks: The most persistent failure mode across all workflows is transition quality (average score 0.256) and temporal coherence (0.405). Even the best workflow (Kling) scored only 0.356 on transitions. In contrast, single-frame fidelity and prompt fulfillment were significantly higher (0.71).
- Architectural Trade-offs: Closed-source agents (Dreamina, Kling) outperformed open-source ones. Dreamina excelled in prompt compliance but struggled with cross-shot continuity due to its incremental edit-action workflow. Kling maintained better global coherence via a shared canvas but sacrificed strict constraint preservation.
- Video Type Sensitivity: Action videos were the hardest category to generate (0.455), while cinematic shot-design was the strongest (0.509).
Base LLM Impact (RQ2):
- Base LLM choice primarily affected narrative reasoning (Script) and cross-modal coordination.
- Visual quality, audio fidelity, and stability were largely governed by the workflow architecture and remained insensitive to the base LLM choice.
- No single LLM dominated all dimensions; models showed complementary specializations (e.g., Seed 2.0 Pro for script, GLM-5.1 for audio).
Personalization Necessity (RQ3):
- Generic evaluation (equal-weight aggregation) masks significant user-dependent variation. The same video could receive scores varying by over 10 percentage points depending on the user profile (e.g., a "Visual-Heavy" user vs. an "Audio & Emotion" user).
- A single aggregate score fails to capture whether a system satisfies specific user intents.
Human Alignment:
- A human evaluation with 14 annotators validated DirectorBench. Human raters agreed that cross-shot continuity and multi-modal consistency were the weakest aspects of current systems, aligning with the benchmark's bottleneck detection.
- Humans also confirmed that preference-dependent quality variations exist, supporting the need for personalized evaluation.

Significance and Claims

The paper claims that DirectorBench addresses a critical gap in long-form video generation evaluation by moving beyond aggregate scores to provide diagnostic, profile-aware insights.

Diagnostic Value: It reveals that the primary bottleneck in current long-form generation is not within individual frames but in the continuity between units (shots and modalities).
Workflow vs. Model: The study demonstrates that workflow architecture is a more dominant factor in quality than the choice of the base LLM, suggesting optimization priorities should focus on system design (e.g., global state propagation) before model selection.
User-Centricity: The benchmark proves that "quality" is not an absolute scalar but a function of user intent. Personalized evaluation is necessary to uncover failure modes and quality variations that generic metrics obscure.

The authors position DirectorBench as a tool to guide targeted method improvements by localizing specific bottlenecks, rather than merely ranking systems. They acknowledge limitations regarding reliance on model-based evaluators and suggest future work should focus on closing the loop by using these diagnostic signals to trigger targeted regeneration or repair mechanisms.