Technical Summary: TriMotion

Problem Statement

Camera motion control is a fundamental technique for directing viewpoint changes, organizing cinematic space, and realizing user intent in generative video systems. While recent advances in conditional video diffusion models have enabled various conditioning signals, existing camera-control methods are typically restricted to a single specific input modality.

• Pose-conditioned methods require explicit geometric camera trajectories, which are unintuitive for general users and difficult to specify accurately.
• Reference-video-based approaches implicitly encode motion patterns but lack explicit trajectory control and offer limited flexibility.
• Text descriptions, while intuitive, have been underexplored for camera control due to a lack of explicit temporal structures and geometric constraints, making precise, physically consistent trajectory execution difficult.

This modality-specific restriction limits the flexibility of generative systems. There is a need for a shared motion representation that bridges heterogeneous inputs (video, pose, and text) to enable consistent camera motion control across different input types.

Methodology

The authors propose TriMotion, a unified, modality-agnostic framework for camera-controlled video generation. The framework maps diverse control signals—video, pose, and text—into a single continuous motion embedding space.

1. Motion Triplet Dataset

To train a cross-modal space requiring synchronized supervision, the authors constructed the Motion Triplet Dataset by extending the Multi-Cam Video Dataset.

• Source: 136K high-fidelity Unreal Engine 5 videos with accurate ground-truth camera extrinsics.
• Processing: An automated pipeline converts raw camera extrinsics into geometry-grounded natural language descriptions.
  • Raw pose sequences are converted into relative trajectories anchored at the first frame.
  • Frame-wise translation and rotation changes are computed and mapped to canonical camera operations (e.g., Dolly, Pan, Tilt) to form compact symbolic sequences.
  • A Large Language Model (Qwen3-4B-Instruct) generates two levels of descriptions: a short summary and a detailed paragraph describing temporal transitions and speed changes, avoiding explicit metric values.
• Result: A dataset providing aligned supervision over videos, camera trajectories, and motion descriptions.

2. Unified Motion Embedding Space

TriMotion learns a shared representation where video, text, and pose inputs are aligned.

• Encoders: Modality-specific encoders ($\Phi_{video}, \Phi_{text}, \Phi_{pose}$) produce sequences of motion tokens.
  • Video: Uses a feature aggregation module (based on VGGT) to extract camera tokens from multi-view geometric information.
  • Text: Uses a frozen T5 encoder with learnable motion queries that cross-attend to text tokens to infer temporal dynamics.
  • Pose: Projects flattened camera extrinsic matrices via a frame-wise MLP.
• Alignment Objectives: The encoders are trained with a composite loss to ensure:
  • Global Alignment: InfoNCE loss aligns global motion tokens across modalities.
  • Temporal Synchronization: Cosine distance minimization between corresponding temporal tokens.
  • Geometric Fidelity: A shared pose regressor predicts camera extrinsics from embeddings, ensuring the space preserves physical 3D geometry.

3. Latent Motion Consistency

To guarantee strict geometric consistency without the cost of pixel-space decoding, TriMotion introduces a Latent Motion Consistency objective.

• Motion Embedding Predictor ($\mathcal{M}_{pred}$): A module consisting of 3D convolutions and a temporal Transformer that estimates a motion embedding sequence from a clean latent video representation ($z_0$).
• Training Strategy: The predictor is trained to align its output with the target motion embedding derived from the ground-truth video.
• Diffusion Integration: During diffusion training, the model reconstructs a clean target latent from the noisy input. This reconstructed latent is fed into the frozen predictor. The predicted motion embedding is then compared to the target motion embedding using a motion consistency loss. This enforces the generated trajectory to match the target directly in latent space.

4. Generation Backbone

The framework utilizes a latent video diffusion transformer.

• Conditioning: The denoiser is conditioned on a global appearance description, the first source frame, and the unified motion embedding sequence injected via block-specific projection MLPs.
• Training: The model supports both Image-to-Video (I2V) and Video-to-Video (V2V) via balanced joint training, updating only the attention layers and projection MLPs to preserve pretrained generative priors.

Key Contributions

1. TriMotion Framework: A unified approach enabling camera control via video, pose, or text inputs by mapping them to a shared motion embedding space.
2. Motion Triplet Dataset: A new dataset pairing multi-view videos with geometry-grounded text descriptions, enabling synchronized multi-modal supervision.
3. Latent Motion Consistency: A novel objective that regularizes the diffusion backbone in latent space, ensuring trajectory fidelity without repeated pixel-space decoding.
4. Cross-Modal Capabilities: The shared embedding space enables novel applications such as sequential motion composition (combining motions from different modalities) and cross-modal motion interpolation.

Experimental Results

Extensive experiments were conducted on I2V and V2V tasks using the Koala-36M and RealEstate10K datasets.

• Quantitative Performance: TriMotion achieves state-of-the-art or competitive results across visual quality (FVD, FID, CLIP Score), dynamic quality (Motion Smoothness, Temporal Flickering), and camera motion accuracy (Rotation Error, Translation Error, CamMC). Notably, the pose-conditioned variant performs best on most metrics, while video and text variants remain highly competitive, validating the effectiveness of the shared space.
• Qualitative Comparison: TriMotion generates cleaner videos with more stable motion and follows target trajectories more faithfully than baselines like CamI2V, MotionClone, and ReCamMaster. It avoids the artifacts and texture degradation seen in methods relying on explicit 3D point-cloud rendering (e.g., TrajectoryCrafter).
• Ablation Studies:
  • Removing the latent motion consistency loss significantly degrades camera accuracy.
  • Target-only conditioning outperforms concatenation or subtraction of source/target embeddings, providing the cleanest motion signal.
  • The unified motion space demonstrates strong cross-modal alignment (high Recall@1) and geometric fidelity (competitive pose regression accuracy).
• User Study: Human evaluations confirm that the geometry-grounded captioning pipeline produces high-quality descriptions and that TriMotion generates videos that follow target motions more faithfully with fewer artifacts than prior methods.

Significance and Claims

The paper claims that TriMotion establishes new benchmarks for camera-controllable video generation by overcoming the limitation of modality-specific inputs. By learning a shared motion embedding space, the framework enables flexible and consistent camera control across heterogeneous inputs.

The authors emphasize that the latent motion consistency objective is crucial for improving trajectory fidelity while bypassing traditional decoding bottlenecks. Furthermore, the shared space pioneers novel interaction paradigms, successfully enabling sequential motion composition and cross-modal motion interpolation without retraining or manual trajectory design.

The paper concludes modestly, acknowledging that while TriMotion effectively transfers smooth camera motions, extreme trajectories requiring large unseen regions may still produce boundary outpainting artifacts. Future work is suggested to integrate explicit 3D background outpainting or extend the framework to jointly model other cinematic attributes like color grading.