SmartDirector: Keyframe-Conditioned Cinematic Video Generation with Narrative Pacing Control

Technical Summary: SmartDirector

Problem Statement
While recent advancements in video generation have shifted from synthesizing short, single-shot clips to creating long, multi-shot narratives, existing methods suffer from a lack of precise control over narrative structure and temporal pacing. Current approaches predominantly rely on sparse conditioning signals, such as text prompts or only the first and last frames. This limitation restricts the ability to manage fine-grained spatial-temporal content. Furthermore, naive strategies that treat adjacent keyframes as start/end points for short clips often result in abrupt temporal discontinuities and a loss of narrative consistency. A more direct approach—inserting keyframes into noisy latents at arbitrary temporal positions—is fundamentally hindered by the causal structure of standard 3D Variational Autoencoders (VAEs). In these VAEs, the first frame is encoded independently while subsequent frames are encoded in groups with causal dependence on preceding frames; arbitrary latent replacement violates this dependency, producing visual artifacts and temporal discontinuities.

Methodology
The paper proposes SmartDirector, a unified, two-stage framework designed for keyframe-conditioned cinematic video generation. The framework supports diverse scenarios, including single-shot generation, multi-shot narrative synthesis, and video extension.

1. Director-Gen (Generation Stage):

   • Multi-Chunk VAE Strategy: To circumvent the causal constraints of the temporal VAE, the video is partitioned into multiple chunks at keyframe positions. Each keyframe serves as the first frame of its respective chunk and is encoded independently. This allows keyframes to be placed at arbitrary temporal positions without violating causal dependencies.
   • Global Consistency: The resulting multi-chunk latents are processed by a Diffusion Transformer (DiT). To maintain global consistency across the entire sequence, the model employs full spatio-temporal attention, enabling each chunk to attend to the global context of all other chunks.
   • MC-RoPE (Multi-Chunk Rotary Positional Embedding): To address temporal discontinuities at chunk boundaries caused by standard positional embeddings, the authors propose MC-RoPE. This assigns fractional temporal indices to keyframe positions, preserving temporal smoothness across the unified latent sequence.
   • Output: This stage generates a low-resolution video (e.g., 480p).

2. Director-SR (Super-Resolution Stage):

   • Keyframe-Conditioned Refinement: Recognizing that low-resolution outputs often lack fine-grained details (e.g., facial features, text), this stage refines the output using high-resolution keyframes as semantic anchors.
   • Mechanism: The module takes the low-resolution (LR) video and high-resolution (HR) keyframes. It employs the Multi-Chunk VAE strategy to encode HR and LR latents independently. At keyframe indices, LR latents are replaced with corresponding HR latents to enforce conditioning.
   • Flow Matching: The model uses flow matching to predict the velocity field mapping the LR latent to the HR latent, effectively upscaling the video to high definition (e.g., 1080p) while correcting generative artifacts.

Data Pipeline
To train the framework, the authors constructed a specialized data pipeline. They curated copyright-free movies, segmented them into single shots, and aggregated consecutive shots with shared scenes and storylines into coherent multi-shot sequences using vision-language models (VLMs). The dataset includes structured captions covering camera motion, character appearances, and holistic narrative descriptions, generated via models like Qwen3-VL and Gemini-3-Pro.

Key Contributions

• Unified Framework: The proposal of SmartDirector, which enables flexible keyframe-conditioned generation for single-shot, multi-shot, and video extension tasks.
• Multi-Chunk VAE Strategy: A novel design that addresses the causal limitations of temporal VAEs, allowing keyframes to be inserted at arbitrary positions while ensuring smooth, continuous generation.
• Keyframe-Conditioned Super-Resolution: A refinement module that utilizes high-resolution keyframes as semantic anchors to recover fine-grained details and correct artifacts, distinguishing itself from standard pixel-level restoration methods.

Results
Extensive experiments demonstrate that SmartDirector substantially outperforms existing state-of-the-art approaches, including the closed-source Dreamina Multiframes system.

• Quantitative Metrics: In single-shot scenarios, SmartDirector reduced the Fréchet Video Distance (FVD) from 226.85 (baseline) to 41.12. In multi-shot scenarios, the improvement was even more significant, reducing FVD from 251.83 to 65.65.
• Semantic Fidelity: Using LLM-based evaluations across five dimensions (Instruction Following, Narrative Coherence, Physical Consistency, Video Quality, and Aesthetic Quality), SmartDirector achieved an average score of 91.30 in single-shot settings and 88.48 in multi-shot settings, significantly outperforming the baseline (83.87 and 59.32, respectively).
• Human Evaluation: In a pairwise Good/Same/Bad (GSB) study, SmartDirector achieved a 54.73% win rate in Overall Quality for multi-shot scenarios, demonstrating superior identity consistency and narrative pacing.
• Super-Resolution Performance: When evaluated independently against SparkVSR, SmartDirector achieved comparable PSNR and SSIM scores but significantly lower LPIPS scores, indicating superior perceptual quality and detail restoration.

Significance and Claims
The paper claims that SmartDirector represents a significant step forward in cinematic video generation by bridging the gap between professional filmmaking practices (storyboarding) and AI generation. By treating keyframes as direct counterparts to storyboards, the framework provides precise control over narrative structure and temporal pacing. The authors emphasize that their Multi-Chunk VAE strategy and MC-RoPE successfully resolve the fundamental conflict between arbitrary keyframe insertion and the causal structure of temporal VAEs. Furthermore, the two-stage design effectively balances computational efficiency with high-fidelity output, enabling the generation of coherent, high-definition multi-shot narratives that maintain entity identity and spatial consistency throughout the sequence. The authors note that while the two-stage pipeline introduces a minor information bottleneck in the initial synthesis, the subsequent super-resolution stage effectively mitigates this, yielding results that approximate the quality of idealized single-stage high-resolution generators.