Technical Summary: OrbitForge

Problem Statement

While generic text-to-video models have evolved into rich open-world scene priors capable of generating high-quality, semantically consistent videos, they do not directly yield reliable 3D assets. Directly converting these videos into 3D scenes faces three primary challenges:

1. Uncontrollable Camera Motion: Camera trajectories in generated videos are implicit, often covering only partial arcs rather than full orbits.
2. Partial View Coverage: Generated videos frequently lack views of the object's back, sides, or ground contact, leading to incomplete 3D reconstructions.
3. Temporal Inconsistency: Frames often contain inconsistencies such as object deformation, texture repainting, and background drift, which, when fitted to 3D Gaussians, result in floaters, blurred geometry, and view-dependent artifacts.

Existing approaches either rely on per-prompt Score Distillation Sampling (SDS) optimization (which is slow and prone to over-saturation), require specialized multiview generators with fixed camera formats (limiting scene context), or focus on local novel-view synthesis without guaranteeing a closed 360-degree orbit.

Methodology: OrbitForge

OrbitForge is an adapter designed to convert a single text-generated video into a canonical, closed-orbit 3D Gaussian Splatting scene without training a new multiview prior or performing per-prompt SDS optimization. The pipeline operates in a reconstruction-render-completion loop:

1. Preliminary Reconstruction as an Anchor

Given a text prompt $p$, a frozen text-to-video model $\Phi$ generates an initial video $V_0$.

• Camera Estimation: An estimator $E$ assigns per-frame cameras $\hat{C}_0$ to $V_0$.
• Deformable Reconstruction: A Deformable Gaussian Splatting (DeformableGS) backend reconstructs the video into a 3D scene $G_0$. This stage uses a deformation field to absorb the "pseudo-dynamics" of the generated video (e.g., object drift, texture changes) rather than baking them into static geometry, thereby organizing the frames into a shared 3D coordinate system.

2. Static Scene Consolidation (MedianGS)

Since the generated video contains non-physical pseudo-dynamics, $G_0$ is converted into a static proxy $G^0_{med}$ to serve as a stable anchor.

• The deformation field is evaluated at $T_m$ uniformly spaced time codes.
• Geometry offsets (position, rotation, scale) are aggregated using a coordinate-wise median across time.
• This robust aggregation extracts the majority deformation state shared by the video, producing a static structural proxy that avoids arbitrary temporal slices or frame-dependent artifacts.

3. Coverage-Aware Closed-Orbit Completion

The static proxy $G^0_{med}$ is rendered along a prescribed canonical orbit $C$ (361 views, 0° to 360°) to identify missing viewpoints.

• Support Analysis: Source cameras are projected onto the fitted orbit plane. The continuous arc supported by the source video ($S_{arc}$) is identified, and the remaining interval ($U_{gap}$) is marked as unsupported.
• Endpoint-Window Completion: Instead of progressive view expansion, the method completes the missing interval by querying the frozen text-to-video model again. The completion is framed as an endpoint-window task: the model fills the "unknown gap" between two known endpoints (the tail and head of the source-supported arc).
• The final video $\hat{V}_0$ consists of the original source frames for $S_{arc}$ and the newly synthesized frames for $U_{gap}$.

4. Second Reconstruction

The completed orbit video $\hat{V}_0$ is reconstructed into the final Gaussian Splatting scene $G_1$ using the prescribed canonical cameras (not re-estimated).

• Preserving the canonical camera system is critical to ensure geometric alignment between the completed views and the final 3D representation.
• The final output is $G_1$, rendered as a static scene $R_1$ on the canonical orbit.

Key Contributions

1. Formulation of Closed-Orbit Generation: The paper defines the task of generating scene-level, closed-orbit 3D assets from generic text-generated videos, explicitly handling uncalibrated cameras, partial coverage, and video inconsistency without task-specific fine-tuning.
2. Reconstruction-Anchored Adapter: OrbitForge proposes a method that uses an imperfect first reconstruction not as a final asset, but as a structural anchor to organize frames and define coverage gaps for completion.
3. MedianGS and Coverage-Aware Completion: The introduction of MedianGS for static scene consolidation and an endpoint-window completion strategy that fills missing orbit intervals globally rather than progressively.
4. Coverage-Aware Evaluation Protocol: The authors argue that local smoothness metrics are insufficient for full-orbit tasks. They introduce a protocol measuring measured angular span, monotone progress, and lower-tail quality (focusing on the hardest, originally unsupported views) to fairly evaluate closed-orbit generation.

Results

Evaluated on a frozen 300-prompt audit set derived from T3Bench:

• Coverage: OrbitForge achieves a measured median span of 359.0 degrees, successfully closing the orbit for 100% of prompts. This significantly outperforms local novel-view methods (e.g., ViewCrafter, which covers only ~18.9 degrees) and matches specialized multiview methods in span while retaining scene context.
• Quality: On the lower-tail quality metric (Q10 ImageReward), OrbitForge raises the score from 8.07 (MedianGS-only reconstruction on unsupported views) to 16.36, demonstrating that the completion stage effectively improves the quality of previously unsupported backside views.
• Comparison: OrbitForge remains competitive with VideoMV on coverage-quality trade-offs but preserves more ground, background, and object-environment relationships compared to object-centric SDS methods or specialized multiview generators.

Significance and Claims

The paper claims that OrbitForge bridges the gap between generative video priors and usable 3D assets by treating the first reconstruction as a scaffold rather than a final product. Its significance lies in:

• Efficiency: It avoids the computational cost and instability of per-prompt SDS optimization and the data requirements of training new multiview generators.
• Scene Fidelity: It leverages the open-world semantic detail of generic video models to produce scene-level 3D assets that include ground contact and background context, which are often lost in object-centric approaches.
• Evaluation Rigor: It establishes that a fair evaluation of 3D generation must prioritize coverage (the ability to render a full orbit) over local smoothness (adjacent-frame consistency), as methods that only generate short trajectories can easily achieve high smoothness scores while failing the core 3D task.

The authors acknowledge limitations: the completed backside is synthesized support, meaning errors in the completion prior (e.g., on transparent or thin structures) can be solidified in the second reconstruction. However, the method provides a robust interface for converting generic text-to-video outputs into closed-orbit 3D Gaussian Splatting scenes.