🤖 AI

DIRECT: Video Mashup Creation via Hierarchical Multi-Agent Planning and Intent-Guided Editing

The paper introduces DIRECT, a hierarchical multi-agent framework that addresses the complex challenge of automated video mashup creation by decomposing the process into screenwriting, directing, and editing stages to ensure professional-grade semantic, visual, and auditory coherence, validated by the new Mashup-Bench benchmark.

Original authors: Ke Li, Maoliang Li, Jialiang Chen, Jiayu Chen, Zihao Zheng, Shaoqi Wang, Xiang Chen

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ke Li, Maoliang Li, Jialiang Chen, Jiayu Chen, Zihao Zheng, Shaoqi Wang, Xiang Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of movie clips—thousands of scenes of car chases, explosions, quiet conversations, and running people. You also have a high-energy song you love. Your goal? To create a perfect, professional-looking music video (a "mashup") where the clips cut together seamlessly, the action matches the beat of the music, and the whole thing tells a thrilling story.

Doing this by hand takes a human editor days of work. Doing it automatically? That's usually a disaster. Most computer programs just grab clips that look like what you asked for, but they end up looking like a messy slideshow where the camera jumps around, the music doesn't match the action, and the story makes no sense.

This paper introduces DIRECT, a new AI system designed to fix this. Instead of using one giant, confused robot to do the whole job, DIRECT acts like a professional film production team, splitting the work into three distinct roles: a Screenwriter, a Director, and an Editor.

Here is how it works, using simple analogies:

1. The Screenwriter: The Architect

The Problem: If you ask a computer to "make a car chase video," it might grab a clip of a car, then a clip of a person eating lunch, then a clip of a rocket ship. It has no sense of structure.
The DIRECT Solution: The Screenwriter is the planner. Before touching a single clip, it listens to your music and reads your request.

What it does: It breaks the song down into sections (Intro, Verse, Chorus). It then looks at your library of clips and groups them into "themes" (e.g., "High-speed chases," "Tense close-ups," "Explosions").
The Analogy: Think of the Screenwriter as an architect drawing a blueprint. They decide: "Okay, the slow, tense intro of the song needs quiet, close-up shots of drivers. The loud, explosive chorus needs fast, chaotic crash scenes." They create a map so the rest of the team knows exactly what to build where.

2. The Director: The Creative Visionary

The Problem: Even with a blueprint, a computer might pick the "right" clips but cut them together in a way that feels jarring. Maybe it cuts from a fast car to a slow car, or switches angles in a way that makes the viewer dizzy.
The DIRECT Solution: The Director takes the Screenwriter's map and gives specific, creative instructions to the Editor.

What it does: It translates the abstract plan into concrete rules. For a "chase scene," it tells the Editor: "Find a motorcycle speeding, but make sure the motion flows smoothly from the previous shot, and cut exactly on the drum beat." For a "sad scene," it says: "Find a close-up of a face, keep the shot long and steady, and don't worry about the beat as much."
The Analogy: The Director is the boss on set. They don't move the camera themselves, but they tell the crew: "We need a smooth transition here, like water flowing. Don't just jump cuts!" They also act as a quality control manager. If the Editor brings back a clip that looks weird, the Director says, "No, that doesn't fit the vibe. Try asking for something slightly different," and sends them back to try again.

3. The Editor: The Precision Craftsman

The Problem: Computers are good at math but bad at "feeling." They might pick a clip that is semantically correct (it's a car) but visually wrong (the car is moving left, but the previous car was moving right, causing a visual jump).
The DIRECT Solution: The Editor is the one who actually assembles the clips, but they do it with superhuman precision.

What it does: The Editor searches the library for clips that match the Director's instructions. But here's the magic: it doesn't just cut at the start of a clip. It looks at the exact frame where the motion matches the previous clip and where the music beat hits. It can trim a 10-second clip down to a 2-second slice to make the transition perfect.
The Analogy: The Editor is like a master watchmaker or a surgeon. They are using a microscopic scalpel to trim the footage. They ensure that when one car scene ends and the next begins, the wheels are spinning in the same direction, and the cut happens exactly when the bass drum hits.

Why is this better than what we have now?

Most current AI editors are like a single person trying to do everything at once. They try to write the story, direct the actors, and edit the film simultaneously. They get overwhelmed and produce a result that is "okay" but lacks soul.

DIRECT works because it separates the thinking from the doing:

Screenwriter plans the big picture (Global Structure).
Director sets the mood and rules for each scene (Local Cohesion).
Editor executes the cuts with mathematical precision (Low-Level Coherency).

The Result

When you use DIRECT, the final video doesn't just look like a random collection of clips. It feels like a professional movie trailer. The camera movements flow naturally, the cuts hit the music perfectly, and the story builds up tension just like a human editor would intend.

The researchers even built a new "test gym" (called Mashup-Bench) to train and test this system, proving that their three-agent team creates videos that humans actually prefer over videos made by other AI tools.

In short: DIRECT turns a chaotic pile of video clips and a song into a cohesive, rhythmic masterpiece by letting an AI team work together just like a real Hollywood crew.

1. Problem Definition

The paper addresses the challenge of Video Mashup Creation, a complex editing paradigm that involves recomposing existing footage from multiple sources into a unified, rhythmically synchronized sequence (e.g., movie montages, AMVs).

Core Challenge: Existing automated frameworks often fail to achieve professional-grade fluidity. They typically rely on a semantic-centric paradigm, treating shots as isolated units selected based on script-guided retrieval. This approach neglects:
- Cross-level orchestration: The interplay between high-level narrative structure, local segment cohesion, and low-level visual/auditory metrics.
- Multimodal nuances: Fine-grained cues like motion dynamics, framing consistency, and precise beat-cut synchronization.
- Result: Generated videos often suffer from disjointed sequences, abrupt visual transitions, and misalignment with the background music.
Formulation: The authors formulate the task as a Multimodal Coherency Satisfaction Problem (MMCSP). This is defined as a joint optimization problem seeking to satisfy three hierarchical constraints:
1. Global Structural Alignment: Aligning narrative flow and visual elements with musical progression (e.g., matching tension in the intro with static shots).
2. Local Segment Cohesion: Ensuring segments reflect adaptive editing intents (e.g., rapid cuts for action vs. stable shots for emotion).
3. Low-Level Coherency: Satisfying explicit metrics for visual continuity (motion, framing) and auditory alignment (beat-sync, energy correspondence).

2. Methodology: The DIRECT Framework

To solve the MMCSP, the authors propose DIRECT (Dynamic Intent for Retrieval & Editing for Cinematic Transitions), a hierarchical multi-agent framework that simulates a professional production pipeline. It decomposes the task into three collaborative modules:

A. Screenwriter (Global Structural Anchoring)

Role: Acts as the structural architect.
Function:
- Footage Summarization: Uses semantic clustering and an MLLM to summarize a massive, unannotated footage library into a structured index of visual themes and clusters.
- Music-Driven Anchoring: Analyzes the music track's structure (intro, verse, chorus) and generates a Global Structural Plan. This plan maps musical sections to specific visual keywords and narrative arcs, ensuring the global structure aligns with the music's emotional progression.

B. Director (Local Cohesion Instantiation)

Role: Acts as the middleware, translating abstract plans into concrete editing guidance.
Function:
- Intent Instantiation: Breaks down global sections into smaller segments. For each segment, it generates three specific constraints:
  1. Semantic Query: Concrete visual subjects (e.g., "motorcycle speeding").
  2. Editing Heuristic: Selects a priority strategy (e.g., Motion Continuity First vs. Framing Consistency First) to weight low-level metrics.
  3. Rhythmic Pacing: Defines shot durations in beats to match the music.
- Closed-Loop Validation: If the Editor fails to find suitable shots, the Director analyzes the failure, relaxes/alters the semantic query (e.g., generalizing "man sprinting in dark alley" to "man running fast"), and retries. This prevents the system from getting stuck due to rigid constraints.

C. Editor (Intent-Guided Sequence Editing)

Role: The executor performing fine-grained optimization.
Function:
- Constrained Path Search: Treats shot selection as a path search problem in a graph. It uses a tailored beam search algorithm to find the optimal sequence of shots that maximizes the weighted sum of low-level metrics (semantic relevance, motion continuity, framing, beat-sync, energy).
- Dynamic Sliding-Window Trimming: Instead of using raw shots, the Editor dynamically trims sub-clips within a sliding window to find the exact frame-level cut-point that maximizes visual continuity (optical flow and saliency matching) while adhering to the rhythmic pacing.
- Pre-computed Features: Relies on pre-extracted features (CLIP embeddings, optical flow, saliency maps) to ensure computational efficiency during the search.

3. Key Contributions

MMCSP Formulation: A novel problem definition that unifies high-level perceptual goals (structure, intent) with low-level explicit metrics into a single optimization objective.
DIRECT Framework: A hierarchical multi-agent system (Screenwriter, Director, Editor) that bridges the gap between high-level planning and low-level execution, incorporating a closed-loop validation mechanism to handle retrieval failures.
Mashup-Bench: A new comprehensive benchmark containing 38 videos (4,000 mins, 64k shots) from 15 movie series, 10 music tracks, and 40 test cases. It includes tailored metrics for visual continuity and auditory alignment, addressing the lack of standardized evaluation in this domain.
State-of-the-Art Performance: Demonstrated significant improvements over existing baselines in both objective metrics and human subjective evaluation.

4. Experimental Results

Objective Metrics: DIRECT significantly outperforms baselines (T2V, MMSC, VideoAgent) in Visual Continuity (Motion Continuity $m_3$ , Framing Consistency $m_4$ ) and Auditory Alignment (Beat-Cut Sync $m_5$ , Energy Correspondence $m_6$ ). While baselines focus on semantic accuracy, DIRECT achieves professional-grade fluidity.
Subjective Evaluation: In a blind study with 60 participants, DIRECT scored highest in Global Structural Alignment and Local Segment Cohesion. Notably, it surpassed the best baseline (VideoAgent) by 40% in Local Segment Cohesion.
Ablation Studies:
- Removing the Screenwriter caused a sharp drop in Global Structural Alignment.
- Removing the Director led to a significant decline in Local Segment Cohesion, proving the necessity of adaptive intent instantiation.
- Removing Dynamic Trimming severely degraded Motion Continuity and Framing Consistency, highlighting the importance of frame-level precision.

5. Significance

The paper represents a paradigm shift in automated video editing from semantic retrieval to multimodal orchestration.

Professionalization: It moves beyond simple "script-to-video" generation, enabling the creation of complex, rhythmically engaging mashups that mimic human editorial intuition.
Scalability & Efficiency: By decoupling heavy visual perception (offline) from the iterative editing loop (online), the system remains efficient despite the complexity of the search space.
Future Impact: The framework provides a robust template for handling other complex multimodal tasks where high-level intent must be translated into precise, low-level physical constraints, paving the way for more sophisticated AI-driven creative tools.