🤖 AI

DramaDirector: Geometry-Guided Short Drama Generation

This paper introduces DramaDirector, a geometry-guided framework that generates visually grounded short dramas by decoupling shots into static and dynamic conditions, leveraging depth-pose retrieval from real cinematography, and utilizing schema-constrained training, alongside the new DramaBoard benchmark for comprehensive evaluation.

Original authors: Hengji Zhou, Sijie Liu, Jianrun Chen, Xingchen Zou, Lianghao Xia, Liqiang Nie

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hengji Zhou, Sijie Liu, Jianrun Chen, Xingchen Zou, Lianghao Xia, Liqiang Nie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to turn a simple story idea into a short, dramatic TV show, like the fast-paced, emotional clips you see on your phone. Doing this with current AI is a bit like asking a writer who has only ever read novels to suddenly direct a movie. They might write a great story, but they don't know the "grammar" of filmmaking—like when to cut the camera, how to frame a fight scene, or how to show two people talking over each other's shoulders.

The paper introduces DramaDirector, a new AI system designed to solve this problem. Here is how it works, broken down into simple concepts:

1. The Problem: The "Novelist" vs. The "Director"

Current AI models are great at writing text, but they struggle with cinematic geometry.

The Issue: If you ask an AI to "show a man and a woman arguing in a garden," it might generate a picture that looks nice but feels wrong. It might put them too far apart, or have them facing the wrong way, because the AI doesn't "see" the spatial rules of a real movie shot.
The Metaphor: It's like giving a chef a recipe that says "make a cake," but the chef has never seen a cake before. They might mix the ingredients, but the result won't look or taste like a real cake.

2. The Solution: Borrowing from a "Visual Library"

Instead of forcing the AI to guess what a movie shot looks like, DramaDirector uses a massive library of real short-drama clips.

The "Blueprint" Approach: The system doesn't just look at the text; it looks at the geometry (the depth and the pose) of real shots. Think of this as a library of blueprints.
How it works: When the AI needs to plan a scene, it doesn't just write a description. It searches its library for a real shot that matches the shape and position of the characters (e.g., "two people standing close, one looking down"). It grabs that "blueprint" (a depth map and a pose skeleton) and uses it as a guide.
The Result: The AI generates the first frame of the video based on this real-world blueprint, ensuring the characters are standing in the right spots and the camera angle makes sense.

3. The Two-Step Process: Planning and Acting

The system splits the job into two distinct roles:

The Planner (The Screenwriter/Director): This part of the AI reads your story and breaks it down into a "storyboard." It decides: "Shot 1: Close-up, eye-level. Shot 2: Wide shot, characters walking." Crucially, it separates the static visual rules (where people stand) from the dynamic story (what they say and do).
The Generator (The Camera Crew): Once the planner picks a "blueprint" from the library, the generator uses it to create the actual video frames. It then animates them to match the story's action.

4. Learning the Rules: "School" and "Practice"

To make the AI a good director, the researchers trained it in two ways:

Supervised Fine-Tuning (SFT): They showed the AI thousands of real storyboards from actual dramas. It's like putting the AI through film school, teaching it the specific rules of short-drama pacing and shot types.
Reinforcement Learning (RL): After school, the AI started "practicing." It generated storyboards, and a special "judge" (another AI) checked: "Does this storyboard match a real shot in our library?" If the AI guessed the geometry right, it got a reward. If it guessed wrong, it learned to do better. This is like a student taking practice tests and getting graded until they master the material.

5. The New Benchmark: "DramaBoard"

The researchers realized there was no good way to test if an AI could actually make these dramas. So, they built DramaBoard.

What is it? A massive dataset containing 35 real dramas, 2,800 episodes, and 81,000 individual shots.
Why it matters: It's like a standardized test for AI directors. It checks not just if the video looks pretty, but if the story makes sense, if the camera cuts are logical, and if the characters stay consistent.

6. The Results

When tested against other top AI systems, DramaDirector won.

Better Stories: The storyboards it created were more logical and followed the "rules" of short dramas better.
Better Visuals: The videos it generated were more consistent (characters didn't morph into different people) and looked more like real movie shots because they were grounded in real geometric blueprints.
More Control: It followed instructions much better. If you asked for a specific camera angle or character position, it actually did it, whereas other systems often ignored those details.

Summary

DramaDirector is like a smart assistant that doesn't just write a story; it knows how to film it. By borrowing real-world camera angles and body positions from a library of actual movies, it bridges the gap between a text idea and a visually coherent video, ensuring the final product looks and feels like a real short drama.

Technical Summary: DramaDirector: Geometry-Guided Short Drama Generation

Problem Statement

The generation of short dramas from plot summaries presents unique challenges that current text-to-video pipelines struggle to address. Short dramas are characterized by rapid shot rhythms, dialogue-driven focus shifts, and specific cinematographic conventions (e.g., over-the-shoulder shots, stacked close-ups) that differ significantly from novelistic storytelling.

Existing approaches face two primary limitations:

Divergence from Shot Grammar: Large Language Models (LLMs), lacking exposure to large-scale short-drama corpora, tend to generate storyboards in a novelistic style. They fail to capture the fast cuts and specific emotional rhythms inherent to short-drama production.
Underspecification of Cinematographic Geometry: Textual descriptions often lack the spatial constraints required to define a frame (e.g., shoulder occlusion, gaze direction, depth ordering). Since these details are inherently visual, downstream generators receiving only ambiguous textual cues cannot recover the necessary geometric fidelity, even with stronger generators or richer prompts.

Methodology: DramaDirector

The authors propose DramaDirector, a geometry-grounded framework that bridges the gap between narrative planning and visual realization by borrowing cinematographic geometry from a gallery of real short-drama shots. The framework operates through a structured pipeline:

1. Geometry-Grounded Generation Pipeline

The system decomposes the generation task into planning and realization stages:

Storyboard Planning: An LLM planner ( $g_\phi$ $g_{ϕ}$ ) generates a storyboard $Y$ $Y$ consisting of pairs of static visual conditions ( $q_j$ $q_{j}$ ) and dynamic narrative conditions ( $d_j$ $d_{j}$ ).
- $q_j$ includes camera framing, scene, and subject state, anchoring the keyframe.
- $d_j$ includes action, dialogue, and emotion, driving the temporal rollout.
Retrieval-Augmented Grounding: Instead of relying solely on text, the system retrieves a depth–pose reference ( $I^r_j$ ) from a pre-indexed gallery ( $G$ ) of real short-drama shots. This retrieval is based on geometric compatibility between the text condition $q_j$ and the gallery shots, measured by a learned text-visual alignment model.
Synthesis:
- First-Frame Generation: A generator ( $F_\xi$ ) synthesizes the keyframe ( $I^0_j$ ) by conditioning on $q_j$ while inheriting layout, pose, and depth from the retrieved reference $I^r_j$ .
- Video Synthesis: An image-to-video model ( $H_\psi$ ) rolls out the shot using $I^0_j$ as the visual anchor and $d_j$ as the temporal instruction.

2. Training Strategy

The planner is trained using a combination of Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL):

Schema-Constrained SFT: The planner is adapted via multi-task supervised finetuning on real short-drama storyboards. Tasks include:
- Storyboard Continuation: Autoregressive prediction of shots.
- Single-shot Prediction: Local realization improvement.
- Multi-field Completion: Recovering missing fields.
- Shot Ordering: Strengthening temporal coherence.
- Reconstruction: Reinforcing plan-to-shot grounding.
RL with GRPO: The planner is further optimized using Group Relative Policy Optimization (GRPO) with a multi-objective reward function:
- Visual Retrieval Reward ( $R_{ret}$ ): A contrastive reward ensuring the generated storyboard's static conditions align geometrically with the target episode's visual vectors.
- Video-Generation Readiness Reward ( $R_{vid}$ ): An LLM judge scores the dynamic narrative conditions for concreteness, coherence, and pacing.
- Schema Validity Reward ( $R_{fmt}$ ): Penalizes missing fields or unparsable outputs.
- The weights for these rewards are linearly annealed to prioritize schema validity and visual grounding before shifting focus to video readiness.

3. Geometry-Aware Text-Visual Alignment

A dedicated alignment model is trained to score the geometric match between a static text condition and a candidate shot.

Input: Text conditions are decomposed into camera, scene, and subject states. Visual evidence consists of depth maps and human-pose maps, stripping away appearance and identity to focus purely on geometry.
Training: The model uses bidirectional contrastive learning with hard negative mining. Negatives are drawn from non-adjacent shots, different episodes, and specifically from within the same drama to avoid trivial mismatches, ensuring the model learns fine-grained geometric distinctions.

Key Contributions

DramaDirector Framework: A novel approach that decouples static visual and dynamic narrative conditions, retrieving depth-pose references from real shots to guide first-frame generation and video synthesis.
Training Paradigm: A hybrid training strategy combining multi-task schema-constrained SFT with GRPO-based RL, utilizing a learned text-visual alignment reward to optimize storyboards for real-shot executability.
DramaBoard Benchmark: A comprehensive dataset constructed from 35 live-action dramas (2.8K episodes, 81K shots) featuring structured storyboards, aligned keyframes, and multi-dimensional evaluation protocols covering narrative quality, instruction following, and intrinsic video quality.
Performance Gains: Empirical evidence demonstrating improvements over representative multi-agent and video generation baselines in faithfulness, consistency, and controllability.

Experimental Results

Experiments were conducted on the DramaBoard benchmark, comparing DramaDirector against eight baselines (including SkyScript, MovieAgent, and SkyReels-A2).

Storyboard Narrative Quality: DramaDirector achieved the highest scores in Relevance (0.7230) and Coherence (0.7732), outperforming baselines like UniVA and MovieAgent. This indicates that multi-task supervision and reward optimization effectively capture short-drama pacing and continuity.
Instruction Following: The model achieved the highest Similarity (0.6826) and Purity (0.9752), demonstrating strong alignment with dynamic narrative prompts and a reduction in hallucinated elements.
Video Intrinsic Quality: DramaDirector attained the highest Aesthetics (0.8189) and the lowest DIS (0.1549, indicating better temporal stability), while ranking second in Consistency (0.9003).
Ablation Studies:
- Removing the retrieval reward caused the largest drop in coherence.
- Depth maps were identified as the dominant retrieval cue; removing them nearly collapsed retrieval performance.
- The combination of depth and pose proved essential for balancing spatial layout and character dynamics.
Retrieval Discriminability: DramaDirector's alignment model outperformed general-purpose vision-language encoders (e.g., Tongyi-Vision) and other RAG methods (e.g., VidoRAG) in retrieval metrics (R@10, N@10), confirming the value of explicit depth-pose projection.

Significance and Claims

The paper claims that DramaDirector successfully transforms cinematographic grounding from a text-prompting problem into a retrieval-and-reward problem. By leveraging real-world geometric priors (depth and pose) rather than attempting to verbalize complex spatial constraints, the framework bridges the gap between high-level planning and low-level visual synthesis.

The authors position this work as a step toward coherent keyframe generation and cinematic rhythm in AI video synthesis. They emphasize that their approach addresses the specific "shot grammar" of short dramas, which generic text-to-video models often miss. The introduction of DramaBoard is highlighted as a critical contribution to the field, providing the first structured benchmark with shot-level annotations and evaluation protocols specifically for short-drama generation.

The paper concludes modestly, acknowledging that while retrieval-augmented grounding improves visual fidelity, it may still struggle with highly complex or long-tail interactions where the reference gallery lacks comparable spatial layouts. Future work is suggested to explore broader genres, tighter audio-visual synchronization, and interactive storyboard editing.