Technical Summary: MoCam - Unified Novel View Synthesis via Structured Denoising Dynamics

1. Problem Statement

Generative novel view synthesis faces a fundamental dilemma in reconciling geometric priors with appearance priors.

• Geometric Priors: Methods relying on reconstructed geometry (e.g., depth maps, point clouds) provide essential spatial alignment for target camera trajectories. However, under significant view changes or monocular input, these priors inevitably become sparse, incomplete, and erroneous due to disocclusion and depth inaccuracy.
• Appearance Priors: Source images or videos offer high-fidelity visual details but lack geometric correspondence with novel viewpoints.
• The Conflict: Existing approaches either propagate geometric errors throughout the generation process (by relying solely on scaffolds) or attempt to fuse geometry and appearance statically. Static fusion leads to signal conflicts where strong appearance cues cause geometric drift early in generation, or flawed geometry permanently "bakes" structural errors into the output late in generation. This results in geometric collapse, visual artifacts, and instability, particularly in 4D re-camera tasks (dynamic scenes) and single-image 3D reconstruction.

2. Methodology: MoCam

MoCam introduces a framework that unifies static and dynamic view synthesis by employing structured denoising dynamics within a diffusion process. The core insight is that diffusion models have distinct representational needs across different denoising phases: early stages require coarse structural anchoring, while later stages demand high-frequency refinement.

2.1 Pipeline Overview

The method operates in three main stages:

1. Scaffold Generation: A dynamic point cloud is constructed from the monocular input video (or a single image replicated for $N$ frames) using a depth estimator and inverse perspective projection. This point cloud is rendered along the target camera trajectory to create a coarse "scaffold video" ($x_{ren}^{tgt}$). This scaffold is spatially aligned with the target motion but contains significant artifacts (holes, distortions).
2. Dual-Conditioning Architecture: The model utilizes two distinct conditioning signals:
   • Scaffold Condition ($c_{ren}$): Derived from the rendered scaffold video, providing strong motion and structural priors.
   • Reference Appearance Condition ($c_{src}$): Derived from the original source video, providing high-fidelity texture and object dynamics.
3. Stage-Wise Generation: Instead of fusing these signals statically, MoCam temporally decouples them based on the denoising timestep $t$.

2.2 Structured Denoising Dynamics

The generation process is governed by a time-dependent context $c(t)$, defined by a switch at a pre-defined threshold $T_{switch}$ (empirically set to 0.85):

$$c(t) = \begin{cases} c_{ren} & \text{if } t > T_{switch} \quad (\text{Early Stage}) \\ c_{src} & \text{if } t \le T_{switch} \quad (\text{Later Stage}) \end{cases}$$

• Early Stage ($t > T_{switch}$): The model conditions solely on the scaffold ($c_{ren}$). Since the latent variable is mostly noise, the primary task is to establish global structure and motion coherence. The model is forced to adhere to the target camera trajectory, deliberately tolerating the geometric incompleteness of the scaffold to anchor the scene layout.
• Later Stage ($t \le T_{switch}$): The model switches to conditioning solely on the source appearance ($c_{src}$). By this stage, a coherent low-frequency structure is already established. The model uses the high-fidelity reference video to synthesize details, refine textures, and actively correct the geometric errors and fill disoccluded regions inherited from the scaffold. This prevents the scaffold's flaws from being permanently baked into the final output.

The framework is built upon pretrained latent video generative models (e.g., Wan2.2), operating in a compressed latent space via a Variational Autoencoder (VAE).

3. Key Contributions

1. Identification of Signal Conflict: The paper identifies the fundamental tension between geometric and appearance priors in generative view synthesis, arguing that static fusion leads to signal interference and error propagation.
2. Structured Denoising Dynamics: The authors propose a principled solution that temporally decouples these signals. By aligning the scaffold with early structural anchoring and the appearance with late-stage refinement, the method achieves robust geometry-appearance disentanglement.
3. Unified Framework: MoCam presents a single framework applicable to both single-image 3D reconstruction and video 4D re-camera. It demonstrates that stage-wise conditioning generalizes across input modalities, addressing the challenge of synthesizing views under unreliable geometry without requiring per-scene optimization.

4. Experimental Results

The method was evaluated on the OpenVid-1M dataset (in-the-wild monocular videos) and the iPhone multi-view dataset.

• Quantitative Performance: MoCam significantly outperforms state-of-the-art methods (GEN3C, TrajCrafter, ReCam) across multiple metrics:
  • Perceptual Quality: Achieves superior scores in VBench metrics (Background Consistency, Subject Consistency, Imaging Quality) and lower FVD-V scores (e.g., 255.16 vs. 289.37 for GEN3C in 3D reconstruction).
  • Pose Accuracy: Maintains the lowest rotation and translation errors, demonstrating precise camera control.
  • Pixel-wise Metrics: On the iPhone dataset, MoCam achieves the highest PSNR, SSIM, and lowest LPIPS and FVD, indicating high fidelity to ground truth.
• Robustness to Geometric Degradation: Experiments varying camera motion magnitude (30° to 90° orbits) show that MoCam maintains consistent performance as geometric sparsity increases. Competitors suffer from catastrophic drift or structural collapse under large view changes, whereas MoCam's stage-wise approach prevents drift and corrects sparse errors.
• Ablation Studies:
  • Scaffold-Only: Results in collapsed metrics, confirming that persistent geometric conditioning bakes errors.
  • Scaffold-Early: Avoids worst artifacts but produces blurry details in disoccluded regions without the late-stage appearance signal.
  • Static-Both: Suffers from catastrophic geometric instability (high rotation/translation errors) due to signal interference, validating the necessity of temporal decoupling.
  • Generalization: The method performs consistently across different backbone architectures (Wan2.1 and Wan2.2).

5. Significance and Claims

The paper claims that MoCam addresses the underlying challenge of synthesizing views under unreliable geometry by transforming the denoising process into a structured progression from alignment to realism.

• Robustness: The method achieves robust geometry-appearance disentanglement, generating coherent and photorealistic results even when point clouds contain severe holes or distortions.
• Error Correction: Unlike prior methods that propagate errors, MoCam's late-stage appearance conditioning actively corrects geometric inaccuracies.
• Unified Approach: It highlights that the same generation principle applies to both static and dynamic view synthesis, offering a unified solution for 3D reconstruction and 4D re-camera without the need for dense multi-view data or per-scene optimization.

The authors conclude that their stage-wise design sets new standards for controllable view synthesis by effectively mitigating the limitations of imperfect monocular reconstructions.