Technical Summary: EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidance

Problem Statement

Recent advancements in video diffusion models (VDMs) have improved realistic video generation, yet controlling camera trajectories remains a significant challenge. Existing approaches typically rely on creating "anchor videos" (rendered videos approximating desired camera motions) to guide diffusion models. These methods generally involve lifting a source image into a 3D point cloud and re-rendering it along a camera trajectory. However, this process faces two critical limitations:

1. Inaccuracy and Misalignment: Errors in point cloud reconstruction and camera trajectory estimation lead to pixel-level misalignments between the anchor and source videos. This forces the model to compensate for rendering errors and repair misaligned content, resulting in inefficient training, high computational costs, and lower quality.
2. Data and Estimation Constraints: High-quality 3D data often requires expensive motion-capture systems, or relies on inaccurate estimators. Consequently, training is often restricted to static, multi-view datasets (e.g., RealEstate10K), limiting generalization to real-world dynamic ("in-the-wild") videos. Furthermore, many prior methods require extensive full fine-tuning of the backbone model, increasing parameter counts and training time.

Methodology

The authors propose EPiC (Efficient and Precise Video Camera Control), a framework designed to learn camera control efficiently by constructing well-aligned training anchor videos without requiring camera pose or point cloud estimation during training.

1. Visibility-Based Anchor Video Construction

Instead of rendering from estimated 3D point clouds, EPiC constructs anchor videos by masking the source video based on first-frame visibility.

• Pixel Tracking: The method estimates pixel trajectories from the first frame using dense optical flow (RAFT).
• Masking: For each subsequent frame, only pixels that can be reliably traced back to the first frame are preserved. Pixels with no valid correspondence (representing newly revealed regions or occlusions) are masked out.
• Artifact Injection: To bridge the gap between training (masked source) and inference (potentially imperfect point cloud rendering), the authors inject synthetic "flying-pixel" artifacts into the masked anchor videos during training. This teaches the model to ignore such artifacts during inference.
• Benefit: This process ensures precise alignment in visible regions and mimics the geometric properties of anchor videos (excluding new content) without needing 3D reconstruction.

2. Anchor-ControlNet

EPiC introduces Anchor-ControlNet, a lightweight module that integrates anchor video guidance into pre-trained video diffusion models (specifically CogVideoX-5B-I2V) without modifying the backbone.

• Architecture: It is a DiT-based module with only ~26M parameters (<1% of the backbone). It injects control signals into the first 25% of the backbone layers using a reduced hidden dimension (256 vs. 3072).
• Visibility-Aware Output Masking: A key innovation is that the ControlNet's output is masked by the visibility map. The ControlNet is tasked only with copying visible content from the anchor video. The synthesis of occluded or invisible regions is entirely delegated to the frozen base diffusion model. This division of labor prevents errors in invisible regions from corrupting the output and simplifies the learning task.

3. Training and Inference

• Training: The model is trained on 5,000 in-the-wild videos (from Panda70M) for only 500 steps. The backbone remains frozen; only the Anchor-ControlNet is trained.
• Inference (I2V): Supports two modes:
  • Full Point Cloud: Standard rendering from estimated depth.
  • Masked Point Cloud: Users can segment dynamic objects (e.g., people, animals) using GroundedSAM. These regions are excluded from the 3D point cloud rendering, allowing the background to drive camera motion while the foreground objects move naturally within the generated video.
• Inference (V2V): Supports video-to-video control by constructing dynamic point clouds from the input video using depth estimation (DepthCrafter) and applying the target trajectory as a relative transformation.

Key Contributions

1. Novel Anchor Video Pipeline: A visibility-based masking strategy that produces perfectly aligned anchor-source pairs without camera pose or point cloud estimation, enabling training on diverse in-the-wild videos.
2. Lightweight Anchor-ControlNet: A visibility-aware adapter that conditions on anchor videos with <1% additional parameters. Its design separates the tasks of copying visible content (ControlNet) and synthesizing invisible regions (Base Model), improving training efficiency and output quality.
3. Efficiency and Generalization: The framework achieves state-of-the-art performance with significantly fewer parameters, training steps, and data compared to prior methods. It demonstrates strong zero-shot generalization to Video-to-Video (V2V) tasks despite being trained solely on Image-to-Video (I2V) data.

Results

• Performance: On the RealEstate10K and MiraData benchmarks for I2V camera control, EPiC achieves the lowest rotation and translation errors (e.g., 0.40 ± 0.11 RotErr on RE10K) and highest total quality scores among compared methods (CameraCtrl, AC3D, ViewCrafter, Gen3C, etc.).
• Efficiency: EPiC converges with 5,000 videos and 500 training steps (approx. 15 GPU hours), which is over an order of magnitude more efficient in terms of data and compute cost than methods like TrajCrafter or ViewCrafter. It uses only 26M trainable parameters compared to billions in some baselines.
• V2V Generalization: Despite training only on I2V data, EPiC achieves performance comparable to methods specifically trained for V2V (e.g., GCD, TrajectoryCrafter) on the Kubric-4D dataset, demonstrating robust zero-shot generalization.
• Qualitative: Visual comparisons show EPiC preserves visible content fidelity better than methods that over-repair misaligned regions and generates more coherent content in invisible regions compared to methods that rigidly follow erroneous anchor signals.

Significance

The paper claims that EPiC addresses the fundamental inefficiency of current camera control methods by transforming the learning task from "repairing misaligned content" to "copying visible regions." By eliminating the dependency on accurate 3D estimation during training and utilizing a lightweight, visibility-aware ControlNet, EPiC enables precise 3D-informed camera control with minimal computational resources. This approach allows for robust application to real-world dynamic videos and supports flexible control over both static and dynamic scene elements, advancing the practicality of controllable video generation for applications like film recapturing and virtual cinematography.