🤖 AI

OmniDirector: General Multi-Shot Camera Cloning without Cross-Paired Data

The paper introduces OmniDirector, a unified framework that enables general multi-shot camera cloning for video generation without requiring cross-paired data by utilizing a novel grid-based camera representation and a hierarchical prompt expansion agent to coordinate characters, actions, and complex camera trajectories.

Original authors: Jiwen Liu, Shujuan Li, Zhixue Fang, Xiaohan Li, Yan Zhou, Zijie Meng, Zhimin Zhang, Yawen Luo, Guoxin Zhang, Yu-Shen Liu, Pengfei Wan

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiwen Liu, Shujuan Li, Zhixue Fang, Xiaohan Li, Yan Zhou, Zijie Meng, Zhimin Zhang, Yawen Luo, Guoxin Zhang, Yu-Shen Liu, Pengfei Wan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a movie director. You have a perfect scene in your head, but you only have a single, still photograph to start with. You want to tell a story, but the camera needs to move: zooming in on a character's face, panning across a landscape, or cutting between different angles.

Usually, asking a computer to do this is like trying to explain a dance to someone who has never seen a dance by only using math equations. It's either too vague (just saying "move the camera") or too complicated (giving exact 3D coordinates that no one can visualize).

OmniDirector is a new tool that solves this by letting you "copy-paste" the camera movements from any video you like onto your own image. Here is how it works, broken down into simple concepts:

1. The "Ghost Room" Trick (The Camera Grid)

The biggest problem with copying camera moves is that computers get confused if the new scene looks different from the old one. If the reference video is a tiny toy car and your image is a real mountain, a direct copy might make the mountain look like a toy.

To fix this, OmniDirector uses a clever trick called the Camera Grid.

The Analogy: Imagine you want to teach someone how to drive a specific route. Instead of showing them a video of a Ferrari driving through Paris (which might confuse them if they are driving a truck in New York), you show them a video of a ghostly, empty room with only grid lines on the floor and walls.
How it works: The system takes the camera moves from your reference video and draws them as a moving grid inside an empty 3D room. This grid shows only the path and angle of the camera, stripping away all the cars, people, and scenery.
The Result: Because the "room" is empty, the computer learns the movement perfectly without getting distracted by the content. It can then apply that exact movement to your mountain, your toy car, or anything else, regardless of size or shape.

2. The "Director's Assistant" (The Prompt Agent)

Once the computer knows how the camera moves, it needs to know what to show. You might have a reference video, a starting photo, and a text description (e.g., "a cat sitting on a fence").

The Problem: If you just throw all these instructions at the computer, it might get confused. It might accidentally copy the cat from the reference video instead of your photo, or it might mix up the camera moves with the story.
The Solution: OmniDirector uses a smart "Assistant" (called a Hierarchical Prompt Expansion Agent).
- The Analogy: Think of this assistant as a translator who speaks "Camera Language" and "Story Language." It looks at the reference video and writes a specific script: "First, the camera pulls back (Shot 1). Then, it cuts to the left (Shot 2)."
- It carefully separates the camera moves from the story details. It ensures the computer knows: "Use the movement from the reference video, but use the cat from your photo." This prevents the computer from accidentally stealing the wrong objects from the reference video.

3. The "One-Stop Shop" (Multi-Shot Cloning)

Most previous tools could only handle one continuous camera move. If you wanted a video with three different shots (like a movie scene), they would fail or get messy.

OmniDirector is special because it handles Multi-Shot videos. It understands that a movie isn't just one long take; it's a series of cuts. It can look at a reference video with multiple cuts and replicate that exact sequence of shots on your image, keeping the story logical and the transitions smooth.

4. Why It's Better Than Before

No "Cheating": Old methods often tried to learn by looking at pairs of videos that were identical except for the camera move. But those videos are hard to find. OmniDirector doesn't need them. It builds its own "training data" by turning millions of random internet videos into those "Ghost Room" grids.
No "Leakage": Because it uses the empty grid and the smart assistant, it doesn't accidentally copy the wrong people or objects from the reference video. It keeps your image clean and only copies the movement.
It Just Works: The paper shows that even if you feed it a raw video or a simple line drawing (like a Canny edge map) instead of the perfect grid, the system is smart enough to figure out the camera moves anyway. It's like a musician who can play a song perfectly even if you hum the tune instead of playing the sheet music.

Summary

OmniDirector is like a magic camera operator. You give it a still photo and a reference video. It strips the reference video down to its "skeleton" (the empty grid movement), uses a smart assistant to write a clear script, and then animates your photo to move exactly like the reference video—whether it's a single zoom or a complex movie scene with multiple cuts. It does this without needing special data or getting confused by the differences in the scenes.

Technical Summary: OmniDirector

Problem Statement

Camera motion is a critical element in video generation, shaping atmosphere, emotion, and narrative depth. However, existing methods for camera control face significant limitations:

Textual Descriptions: While accessible, they fail to precisely define nuanced cinematographic attributes.
Explicit Parametric Representations: Methods using matrices or Plücker coordinates offer precision but impose high user barriers and struggle with multi-shot generation and shot transitions.
Video-Referenced Approaches:
- Explicit Parameter Extraction: Suffers from scale ambiguities when transferring camera trajectories from reference environments to generated scenes with different spatial scales.
- Implicit/End-to-End Learning: Relies on cross-paired data (videos with identical camera motion but different content). Such data is scarce in the real world. While synthetic data can alleviate this, it often lacks the narrative complexity of real footage, leading to poor performance in handling shot cuts and intricate transitions. Furthermore, existing methods often suffer from "information leakage," where the model inadvertently copies appearance or character motion from the reference video rather than just the camera trajectory.

There is a lack of a general framework capable of cloning complex, multi-shot camera motions from reference videos without requiring cross-paired training data or suffering from content leakage.

Methodology

The authors propose OmniDirector, a unified framework designed for Multi-Modal Diffusion Transformers (MMDiTs) to achieve general multi-shot camera cloning. The methodology consists of three core components:

1. Camera Grid Representation

To decouple camera motion from scene content and enable scalable training, the authors introduce the Camera Grid.

Concept: Instead of using raw video or explicit parameters, camera motion is rendered as a video of a 3D empty scene containing a spatial grid.
Construction:
- Camera poses ( $R_i, t_i$ ) are extracted from reference videos.
- An empty 3D scene is modeled with orthogonal grid lines representing the floor, ceiling, and a "tunnel wall" surrounding the camera trajectory to visualize relative movement.
- Special effects like fisheye distortion and dolly zoom (Hitchcock zoom) are supported by modifying the rendering scheme (e.g., using the Kannala–Brandt model for distortion).
Advantages:
- Generality: Unifies single and multi-shot cloning.
- Decoupling: The empty scene prevents interference from appearance or character motion.
- Scalability: Any video can be converted into a camera grid, allowing the construction of million-scale training datasets without the need for manually curated cross-paired data.

2. OmniDirector Architecture & Training

Architecture: Built upon a Multi-Modal Diffusion Transformer (MMDiT).
Input Injection: The camera grid ( $G$ ), reference image ( $I$ ), and noisy video latent ( $Z_v$ ) are encoded via a 3D VAE and concatenated along the frame dimension to form a unified spatiotemporal representation ( $z_{vis}$ ). Text conditions are processed separately and interact with visual tokens via joint attention.
Training Strategy:
- Dataset: Trained on 1.8 million internet-scale video pairs (reference video + generated camera grid).
- Self-Reconstruction Objective: 30% of training samples use the camera grid itself as the target video. This forces the model to learn the geometric structure and temporal dynamics of the grid rather than treating it as a weak hint, preventing overfitting to spurious correlations.

3. Hierarchical Prompt Expansion (PE) Agent

To harmoniously integrate camera control with other signals (user prompts, reference images) during inference, a novel PE Agent is employed:

Camera Prompt Generation: An MLLM (Qwen3-VL) generates a camera prompt ( $T_c$ $T_{c}$ ) decomposed into:
- Inter-shot: Analyzes relationships between adjacent shots to ensure semantic coherence across transitions.
- Intra-shot: Describes camera motion within individual shots based on pose estimation and visual correction.
Semantic Fusion: The agent fuses the camera prompt, reference image, and user prompt into a single, cohesive text condition, preventing conflicts and background leakage.
Adaptive Classifier-Free Guidance (CFG): A coarse-to-fine denoising strategy is used. Camera grid features are injected during the high-noise regime to establish global spatial structure, while other signals are introduced in the low-noise regime to refine local details.

Key Contributions

OmniDirector Framework: A unified video generation framework achieving general multi-shot camera cloning without cross-paired data, providing director-level control for MMDiTs.
Camera Grid: A novel, general visual representation for camera motion that decouples trajectory from content, enabling the construction of million-scale training datasets.
Hierarchical Prompt Expansion Agent: A mechanism that systematically integrates camera motion with multimodal signals, ensuring semantic coherence in multi-shot scenarios and minimizing information leakage.

Experimental Results

The framework was evaluated on a diverse set of 1,094 samples covering single/multi-shot videos, various resolutions, and complex effects.

Quantitative Performance: OmniDirector outperforms state-of-the-art baselines (CamCloneMaster, Seedance2.0, LTX-LoRA) across all metrics.
- Camera Accuracy: Achieved a Relative Translation Error (RTE) of 16.84° and a Translation Precision (T-Pre) of 72.74%, representing a 39.3% relative improvement in translation precision over the second-best method.
- Transition Accuracy: Achieved 96.52% Temporal Precision (Tem-Pre) and 83.79% Semantic Precision (Sem-Pre) for shot transitions.
- Leakage Rate: Demonstrated the lowest leakage rate (0.51% frame, 3.38% shot), significantly outperforming baselines that struggle with content leakage.
Qualitative Performance: The model faithfully clones complex trajectories (including dolly zooms and arc shots) and maintains semantic coherence in multi-shot sequences, whereas baselines often fail at transitions or exhibit motion artifacts.
Emergent Capabilities: The model demonstrates zero-shot generalization, successfully executing camera motions when conditioned on raw RGB videos or Canny edge sequences without retraining, suggesting the camera grid effectively unlocks an understanding of spatial dynamics.

Significance

The paper claims that OmniDirector addresses the fundamental trade-off between precision and accessibility in camera control. By introducing the camera grid, the authors bypass the data scarcity of cross-paired datasets and the scale ambiguity of explicit parameter extraction. The framework enables the generation of high-fidelity, multi-shot videos with director-level control, effectively bridging the gap between intuitive reference-based control and the rigorous requirements of professional cinematography. The work establishes a new paradigm for training video generation models on massive, automatically curated datasets derived from standard internet videos.