💻 computer science

SymphoMotion: Joint Control of Camera Motion and Object Dynamics for Coherent Video Generation

SymphoMotion is a unified framework that jointly controls camera trajectories and object dynamics using explicit geometry-aware cues and 3D trajectory embeddings, supported by a new real-world dataset (RealCOD-25K), to achieve significantly improved visual fidelity and motion coherence in video generation compared to existing methods.

Original authors: Guiyu Zhang, Yabo Chen, Xunzhi Xiang, Junchao Huang, Zhongyu Wang, Li Jiang

Published 2026-06-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Guiyu Zhang, Yabo Chen, Xunzhi Xiang, Junchao Huang, Zhongyu Wang, Li Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a movie director holding a camera. In a real film, you have two big jobs: you have to move the camera around the set (panning, tilting, zooming), and you have to direct the actors to move across the stage. If you move the camera but the actors stand still, it looks weird. If the actors run but the camera stays fixed, you lose the drama.

The problem with current AI video generators is that they are like directors who can only do one of these jobs at a time. Some AI tools are great at moving the camera, but the actors in the video freeze or glitch. Others are good at making objects move, but if you try to move the camera, the objects warp or disappear because the AI gets confused about what is moving and what is just the viewpoint changing.

SymphoMotion is a new AI system that solves this by acting like a conductor of a symphony orchestra. It learns to coordinate the "camera" and the "actors" (objects) at the same time, so they move together in a realistic, synchronized way.

Here is how it works, broken down into simple parts:

1. The Two Special Tools

The system uses two specific "control knobs" to manage the video:

The Camera Control (The "Viewpoint" Tool):
Imagine you are walking through a room. As you walk, the walls and furniture shift in your vision because of your movement. Old AI tools often just tried to "slide" the image, which made the room look flat and fake.
SymphoMotion uses a 3D map (a digital cloud of points representing the room's shape) to understand the room's structure. When you tell the AI to move the camera, it uses this 3D map to ensure the walls and furniture shift realistically, just like they would if you were actually walking through the scene. It keeps the geometry stable so the video doesn't look like a melting painting.
The Object Control (The "Actor" Tool):
Imagine you want a dog to run across a field. Old tools might just tell the AI "move the dog to the right." But if the camera is also moving, the AI gets confused: Is the dog moving, or is the camera moving?
SymphoMotion solves this by giving the AI two sets of instructions for the dog:
1. 2D Instructions: A simple drawing on the screen showing where the dog should be in the picture (like a sticky note on a window).
2. 3D Instructions: A real-world path in 3D space showing exactly how the dog moves through the air and depth.
  By combining these, the AI knows exactly how the dog should move relative to the camera, ensuring the dog stays solid and moves naturally even while the camera spins around it.

2. The Missing Puzzle Piece: A New Dataset

To teach an AI to do this, you need a teacher with a lot of examples. The problem was that no one had a library of real-world videos where both the camera movement and the object movement were carefully measured and labeled. Existing videos either had moving cameras with static scenes, or moving objects with a fixed camera.

The researchers built RealCOD-25K. Think of this as a massive, high-quality library of 25,000 video clips. For every clip, they didn't just save the video; they also saved the "script" for the camera (where it went) and the "script" for the objects (where they went in 3D space). This allowed them to train the AI to understand the complex relationship between moving the lens and moving the subject.

3. How You Use It

The paper describes a user-friendly interface (like a digital art studio):

You upload a single photo.
The AI builds a 3D model of that photo.
You can drag objects in 3D space (like pulling a toy car along a path) to tell the AI how you want them to move.
You can draw a path for the camera to follow.
The AI generates a video where the camera moves along your path, and the objects move exactly where you dragged them, all while looking like a real, coherent scene.

The Result

When tested, SymphoMotion created videos that looked much more realistic than previous methods.

Visual Quality: The videos looked sharper and less distorted.
Camera Control: The camera moved exactly as requested without warping the background.
Object Control: Objects moved smoothly and stayed in the correct place relative to the camera, even during complex movements.

In short, SymphoMotion is the first system that successfully teaches an AI to be a full movie director, capable of handling the camera and the actors simultaneously to create a believable, moving world.

Technical Summary: SymphoMotion

Problem Statement

Controlling both camera motion and object dynamics simultaneously is essential for generating coherent and expressive videos. However, current methods face significant limitations:

Single-Modality Focus: Existing approaches typically handle either camera motion (e.g., CameraCtrl, Uni3C) or object dynamics (e.g., MotionCtrl, MagicMotion) in isolation.
2D Ambiguity: Methods attempting joint control often rely on 2D cues (optical flow, bounding boxes) within the image plane. This fails to disentangle true object movement from camera-induced parallax, leading to unsynchronized behaviors and reduced realism, especially under significant viewpoint changes.
Data Scarcity: There is a critical lack of real-world datasets that provide paired annotations of camera poses and object-level 3D trajectories. Existing datasets are either camera-centric (static scenes) or object-centric (fixed cameras), while synthetic datasets suffer from domain gaps.

Methodology: SymphoMotion

SymphoMotion is a unified, diffusion-based framework designed to jointly govern camera trajectories and object dynamics within a single model. It is built upon the pre-trained Wan-I2V architecture and introduces two complementary mechanisms:

1. Camera Trajectory Control (CTC)

To ensure stable and structurally consistent viewpoint transitions, CTC integrates explicit camera paths with geometry-aware cues:

3D Geometric Priors: Instead of relying solely on Plücker embeddings, the method estimates a point cloud from the reference image using Depth-Pro.
Viewpoint Control Module (VCM): This module fuses two encoded inputs:
- Motion Representation ( $c_{cam}$ ): Derived from Plücker embeddings of the target camera pose sequence.
- Geometry Representation ( $c_{pcd}$ ): Extracted from point-cloud frames rendered along the target trajectory using a Wan encoder.
Integration: These features are concatenated with the noisy latent and added as explicit motion cues to the diffusion model, enhancing spatial coherence and geometric fidelity.

2. Object Dynamics Control (ODC)

To enable depth-aware, spatially coherent object manipulation, ODC combines 2D visual guidance with 3D trajectory embeddings:

2D Visual Guidance: 3D object trajectories are projected onto the image plane based on target camera poses. Per-frame bounding boxes are fitted and rendered onto the point-cloud frames, serving as explicit spatial anchors to guide the model in localizing objects across frames.
3D Trajectory Conditioning: An Object Motion Module (OMM) encodes the 3D trajectories (transformed to the reference camera's coordinate system) and semantic entity prompts. These are fused and injected into the diffusion transformer blocks via cross-attention, allowing the model to align latent representations with specified 3D paths.

Training and Inference

Training Strategy: A two-stage approach is employed. First, the CTC module is trained to learn camera control; subsequently, the CTC is frozen, and the ODC module is trained for object motion.
Inference Pipeline: Users provide a reference image, a text prompt, and interactive inputs. The system reconstructs a 3D point cloud, allows users to select objects (via SAM2) and drag them in 3D space to define trajectories, and specifies a camera path. The model then generates a video adhering to these constraints.

Key Contributions

Unified Framework: The proposal of SymphoMotion, the first framework to jointly control camera motion and object dynamics in a single model, enabling coherent, flexible, and depth-aware motion specification.
RealCOD-25K Dataset: The construction of a large-scale, real-world dataset containing over 25,000 video clips with synchronized camera poses and object-level 3D trajectories. This addresses the critical data gap for training and evaluating unified motion-control models.
Performance: Extensive experiments and user studies demonstrating that SymphoMotion outperforms state-of-the-art methods in visual fidelity, camera controllability, and object-motion accuracy.

Experimental Results

The paper evaluates SymphoMotion against baselines like CameraCtrl, ViewCrafter, Uni3C, and MotionCtrl using a curated set of 100 real-world videos.

Quantitative Metrics:
- Visual Quality: SymphoMotion achieves the lowest FID (70.47) and FVD (332.50), indicating superior image quality and temporal coherence compared to baselines (e.g., MotionCtrl FVD: 738.41).
- Camera Control: It achieves the lowest translation (0.37) and rotation (0.05) errors, outperforming dedicated camera control baselines.
- Object Control: It attains a significantly higher Box-IoU score (61.88) compared to MotionCtrl (31.42), demonstrating more accurate trajectory adherence.
User Study: In a blind study (1-5 scale), SymphoMotion received the highest ratings across all categories: Visual Quality (4.87), Text Alignment (4.02), Camera Motion (4.36), and Object Motion (4.58).
Ablation Studies: Removing 3D geometric priors ( $c_{pcd}$ ), 2D visual guidance (bounding boxes), or 3D trajectory conditioning resulted in measurable performance drops across all metrics, confirming the necessity of each component.

Significance

The paper claims that SymphoMotion establishes a new benchmark for unified motion control in video generation. By successfully disentangling camera-induced parallax from true object dynamics through 3D-aware representations, it enables the generation of videos that faithfully reflect complex, user-defined camera and object movements. The introduction of RealCOD-25K is highlighted as a foundational contribution to the field, providing the necessary supervision to move beyond the limitations of 2D-only control or synthetic-data reliance.