Towards Realistic and Consistent Orbital Video Generation via 3D Foundation Priors

1. Problem Statement

The paper addresses the challenge of generating realistic and consistent orbital videos from a single object image. While existing video generation models (based on diffusion transformers) produce high-quality frames, they struggle with long-range extrapolation (e.g., synthesizing a rear view from a front view).

• Limitations of Current Methods: Existing approaches rely heavily on pixel-wise attention mechanisms. These fail when pixel correspondences between the input and target views are limited or non-existent, leading to geometric inconsistencies, distorted shapes, and unnatural structures in unseen views.
• Limitations of 2.5D Priors: Recent attempts to use 2.5D priors (like depth or normal maps) are insufficient because they cannot model complete object shapes, leaving unobserved or occluded parts under-constrained.

2. Methodology

The proposed method introduces a novel pipeline that leverages 3D foundation priors to guide a base video diffusion model. The core idea is to use the rich structural knowledge learned by large-scale 3D generative models to enforce geometric consistency.

A. 3D Foundation Priors

Instead of relying on intermediate 2.5D representations, the authors utilize a pre-trained 3D foundation model (specifically Hunyuan3D) trained on large-scale 3D asset corpora. This model operates in a native 3D latent space, allowing it to reconstruct complete object shapes directly.
The system extracts two scales of latent features from the input image:

1. Global Latent Vector ($\hat{p}$): A denoised vector representing the overall structural guidance of the object.
2. Local Volumetric Features ($\hat{f}$): A set of latent images projected from volumetric features queried on the global vector. These provide view-dependent and fine-grained geometry details.

• Advantage: Using compact latent features avoids the time-consuming mesh extraction process (e.g., Marching Cubes), ensuring an efficient inference pipeline.

B. Multi-Scale 3D Adapter

To integrate these 3D priors into the base video diffusion model (adapted from Stable Video Diffusion - SVD), the authors introduce a Multi-Scale 3D Adapter.

• Mechanism: The adapter injects the global and local features into the base model via alternating cross-attention layers.
  • The global vector is duplicated across all frames to provide a unified shape reference.
  • The local volumetric features are projected into canonical views and processed through a separate cross-attention layer to handle view-dependent details.
• Design Philosophy: This approach acts as a "soft constraint," allowing the base model to retain its stochastic nature and temporal priors from general video pretraining while being guided by the 3D shape information. It is model-agnostic and does not require retraining the entire base model.

C. Training and Inference

• Training: The model is trained on a large-scale dataset (Objaverse-XL) with synthetic orbital videos. The 3D foundation model and its geometry decoder remain frozen during training; only the adapter and base video model are fine-tuned.
• Inference: The process involves:
  1. Extracting 3D priors from the input image.
  2. Injecting these priors into the video diffusion model via the adapter.
  3. Generating the orbital video following a specified camera trajectory (azimuth and elevation).

3. Key Contributions

1. Novel Integration of 3D Priors: The first method to effectively incorporate 3D foundation priors directly into a video generation pipeline, significantly improving shape realism and multi-view consistency compared to pixel-based or 2.5D approaches.
2. Multi-Scale 3D Adapter: A plug-in module that efficiently injects both global structural guidance and local geometric details into base video transformers via cross-attention, enabling flexible fine-tuning without disrupting pre-trained temporal capabilities.
3. Efficiency: By operating in latent space and avoiding explicit mesh extraction, the method maintains a high inference speed while achieving superior geometric coherence.

4. Results and Evaluation

The method was evaluated on Objaverse-XL and GSO datasets, comparing against state-of-the-art baselines including SV3D, Hi3D, Wonder3D, Era3D, and 3D generation models like Hunyuan3D and Trellis.

• Quantitative Performance:
  • Visual Fidelity: Outperformed baselines in PSNR, SSIM, and LPIPS.
  • Shape Realism: Achieved higher CLIP scores, indicating better semantic alignment with realistic object shapes.
  • Consistency: Demonstrated superior multi-view consistency (measured by the MEt3R metric), significantly reducing artifacts in rear and side views.
• Qualitative Performance:
  • Generated videos exhibit plausible structures and consistent lighting across frames, avoiding the "morphing" or distortion common in baseline video generation methods.
  • Successfully generalized to in-the-wild images and complex dynamic camera trajectories (non-zero elevation angles).
• Ablation Studies: Confirmed that using both global and local features yields the best results. The cross-attention adapter was shown to be superior to simple feature concatenation or frame stacking, as it better preserves the base model's pre-trained capabilities.

5. Significance

This work represents a significant step forward in 3D-aware video generation. By bridging the gap between 3D foundation models and video diffusion, it solves the critical issue of geometric inconsistency in novel view synthesis.

• Applications: The method is highly relevant for e-commerce (product visualization), extended reality (XR), and game design, where consistent, high-fidelity 3D object rotation is essential.
• Future Impact: It establishes a paradigm where 3D structural priors can be used as auxiliary constraints to guide generative video models, potentially paving the way for more robust and controllable 3D content creation without the computational overhead of explicit 3D reconstruction pipelines.