Probing into Camera Control of Video Models

Technical Summary: Probing into Camera Control of Video Models

Problem Statement
Video generation models possess rich priors for spatial structure and temporal dynamics, yet enabling controllable camera motion remains a significant challenge. Existing approaches typically rely on fine-tuning video models on paired camera-pose datasets using additional camera modules (e.g., adapters, embeddings, or attention modifications). The authors identify three fundamental limitations in this paradigm:

1. Implicit and Uninterpretable Mapping: The relationship between camera motion and content is implicitly encoded, often learning dataset-specific correlations rather than underlying geometry.
2. Data Limitations: Available camera-annotated datasets are often limited in scale, diversity, and dynamics (typically synthetic or Structure-from-Motion derived), restricting generalization to realistic motion patterns and complex scenes.
3. Prior Degradation: Fine-tuning on specialized datasets tends to bias the model toward a narrower output distribution, overriding the diverse priors learned during large-scale pretraining and degrading visual quality.

Furthermore, existing training-free methods often rely on non-differentiable operations (e.g., point cloud reconstruction, inpainting), making end-to-end optimization intractable.

Methodology: CamProbe
The paper proposes CamProbe, a training-free method that reformulates camera control not as a generative learning problem, but as a form of geometric guidance applied via displacement fields.

• Displacement Field Formulation: Instead of learning a mapping from camera trajectories to video, the method constructs a set of displacement fields ($\mathcal{F}$) for a given camera trajectory. Given a camera trajectory $T_f = (R_f, t_f)$, the method computes relative transformations with respect to the first frame. It lifts latent pixel coordinates to pseudo-3D points using an off-the-shelf depth estimator ($D_f$), applies the camera transformation, and projects them back to the image plane to derive a dense warping grid (displacement field).
• Differentiable Resampling in Diffusion: These displacement fields are integrated into the denoising process of the video diffusion model. At each timestep $t$ (specifically during the early stage $t \in [T, 0.8T]$), the estimated clean latent $\hat{z}_0$ is warped using the displacement field via a differentiable resampling operator ($\circ$).
  • The update follows a classifier-free guidance form: $\hat{z}'_{0,f} = \hat{z}_{0,f} + \omega (\mathcal{F}_f \circ \hat{z}_{0,f} - \hat{z}_{0,f})$.
  • The modified estimate is used to update the noisy latent $z_t$ (or the velocity $v_t$ in flow matching) before the next sampling step.
• Training-Free Nature: The method requires no additional modules, fine-tuning, or paired data. The displacement fields can be heuristically defined (as done in this work using depth-based warping) or learned.

Key Contributions

1. Reformulation of Camera Control: The authors propose treating camera control as a set of displacement fields applied via differentiable resampling during denoising, enabling control without training or architectural changes.
2. Probe for Model Analysis: The method serves as a standardized probe to isolate and analyze the inherent camera control capabilities of base video foundation models, separating model behavior from artifacts induced by text-conditioning or specific training datasets.
3. Identification of Universal Biases: Using the probe, the authors identify systematic behaviors across representative video models:
   • Underestimation of Capability: Models often perform better under geometric guidance than under text prompts, suggesting limitations lie in the guidance mechanism rather than the model itself.
   • Asymmetric Leakage: Translational motions are easier to generate than rotational ones; rotational commands often leak into translation, but the reverse is less common.
   • Directional Preference: Models exhibit a bias toward horizontal motion over vertical motion.
   • Mode Shift: Excessive control strength can cause a "mode shift" where smooth motion degrades into abrupt transitions.
4. Multi-View Benchmarking: The paper introduces a multi-level evaluation protocol (2D, 2.5D, and 3D) to assess multi-view consistency and benchmarks five video models under orbital camera motions.

Results

• Performance vs. Fine-tuned Baselines: Compared to state-of-the-art fine-tuned methods (e.g., Gen3C, TrajectoryCrafter, ReCamMaster), CamProbe achieves competitive visual quality with only a minor drop in camera control accuracy (measured by Relative Pose Error). Crucially, it preserves the base model's generative prior, resulting in less degradation in imaging quality and motion smoothness compared to fine-tuned approaches.
• Efficiency: The method incurs negligible additional GPU memory overhead. The primary runtime cost comes from depth estimation and latent resampling, which can be reduced by using constant depth or skipping certain resampling steps.
• Probe Findings:
  • Single-View: Models show a trade-off between motion strength and quality. HunyuanVideo-1.5 and Wan2.2-I2V-A14B demonstrate the best balance.
  • Multi-View: While some models (e.g., LTX-2.3) produce high-quality static views, others (e.g., HunyuanVideo-1.5) achieve a better balance between geometric consistency and motion alignment, indicating higher potential for 3D/4D tasks.
  • Comparison with CamTrol: CamProbe produces stronger dynamics than the training-free method CamTrol while maintaining comparable visual quality, attributed to its preservation of intrinsic model dynamics rather than propagating a single static reference.

Significance and Claims
The paper argues that camera control in video generation need not be treated as an implicit learning problem requiring massive datasets and fine-tuning. Instead, it can be effectively solved through displacement field-guided generation. The authors claim that even a heuristically defined displacement field serves as a sufficiently strong signal to induce camera motion without compromising the base model's quality.

The significance of this work lies in two areas:

1. Practical Utility: It offers a simple, training-free mechanism to control camera motion in existing video models, preserving their high-fidelity generative capabilities.
2. Analytical Insight: It provides a tool to "probe" the latent capabilities of foundation models, revealing that many models possess inherent geometric understanding that is often obscured by current evaluation methods (which rely on text prompts) or training paradigms. The findings suggest that the limitations in current camera control are often due to the induction mechanism rather than the models' inability to generate geometrically consistent content.

The authors acknowledge limitations, noting that the current heuristic displacement field may be less reliable in scenes with complex spatial relationships and that the geometric guidance is coarse. They suggest that future work could focus on learning the displacement fields or refining the diffusion update strategies for even greater precision.