Technical Summary: Real2SAM2Real

Problem Statement

Video Diffusion Models (VDMs) have achieved high-fidelity video synthesis by leveraging rich spatiotemporal priors learned from massive datasets. However, enabling precise control over camera trajectories and scene dynamics remains a significant challenge. Existing approaches generally fall into two paradigms, both of which suffer from critical limitations in complex scenarios:

1. Parameterized Camera Embeddings: Methods that inject numerical camera poses (e.g., extrinsic matrices) into the model often struggle with scale ambiguity and a large cross-modal gap between abstract poses and dense visual features, leading to sluggish adaptation and degradation of pre-trained priors.
2. "Warp-and-Inpaint" Strategies: These methods utilize explicit pixel-wise 3D reconstructions (e.g., point clouds) to guide generation. While they reduce the modality gap, they rely on incomplete geometry (unclosed visible shells). This introduces perspective ambiguities (structural holes, erroneous facades) and misleading cues in non-Lambertian regions (reflections, refractions). Crucially, both paradigms fail to provide reliable 3D context for unobserved regions, forcing VDMs to "hallucinate" missing content, which inevitably leads to structural collapse during high-dynamic movements or severe occlusions.

The core issue identified is that VDMs, while robust in 2D temporal and semantic priors, lack the supplementary 3D-aware guidance necessary to navigate complex spatial dynamics without generating artifacts.

Methodology

The authors propose Real2SAM2Real, a framework that leverages generative 3D lifting models to create an explicitly editable, instance-complete 3D geometry cache. This cache serves as a robust geometric scaffold for the VDM, decoupling spatial layout from appearance.

1. Instance-Complete 3D Geometry Cache

Instead of attempting to reconstruct an entire scene (including infinite-depth backgrounds) from a single image, the framework restricts explicit geometric modeling to foreground entities.

• Construction: Starting from a single reference image, a layout-aware 3D generation framework (e.g., SAM3D) extracts a union of decoupled, topologically closed 3D foreground meshes ($O_k$).
• Dynamic Configuration: User-specified spatial manipulations (e.g., 6-DoF rigid transformations) are applied to these instances. Because each instance is a closed 3D asset, the resulting scene configuration ($S^{(t)}_{3D}$) inherently accounts for unobserved back-surfaces, circumventing the "cardboard effect" and structural holes common in monocular warping.

2. 3D-Free Data Curation and Perturbation

To bridge the gap between explicit 3D priors and the VDM's visual domain, the system uses instance-masked normal maps as a cross-modal bridge.

• Representation: Normal maps are dense, scale-invariant geometric descriptors that resemble RGB space, reducing the domain gap.
• Data Pipeline: The authors employ a "3D-free" automated curation strategy. Using off-the-shelf 2D dense prediction (NormalCrafter) and instance segmentation (SAM 3, VLMs), they extract pseudo-normal sequences from monocular video datasets. This avoids the domain gap associated with synthetic 3D datasets.
• Perturbation: To address the discrepancy between clean training data and the noisy, coarse boundaries of single-image 3D lifting during inference, an instance-level random perturbation strategy is applied during training (elastic deformations, morphological operations). This teaches the VDM to treat normal maps as resilient, coarse anchors rather than rigid pixel-wise constraints.

3. Soft Spatial-Aligned Injection and Fine-Tuning

The framework utilizes a minimally invasive fine-tuning strategy on a foundational VDM (Wan 2.2 Image-to-Video DiT) to preserve zero-shot generalization.

• Dual-Path Injection:
  • Appearance ($c_{app}$): Injected via a dual-path mechanism (VAE latents prepended to the sequence; CLIP features via cross-attention).
  • Geometry ($c_{geo}$): Injected via a Soft Spatial-Aligned mechanism. The normal map sequence is encoded by a frozen VAE, processed by a trainable 3D convolutional patch embedding layer, and additively injected into the DiT hidden states.
• Decoupling: Crucially, the geometric condition bypasses the prepended appearance tokens. This prevents the "baking" of coarse geometric boundaries into the texture, avoiding appearance distortion and color bleeding.
• Efficiency: Only ~1.8% of the network parameters (patch embedding layer and LoRA modules) are trainable, while the backbone, VAE, and encoders remain frozen.

Key Contributions

1. Instance-Complete 3D Geometry Cache: The introduction of an explicitly editable, instance-complete 3D proxy that serves as complementary in-context guidance. This formulation unlocks high-dynamic generation capabilities while providing robust spatial anchors that circumvent the perspective ambiguities and artifacts of pixel-level reconstructions.
2. Cross-Modal Bridge via Normal Maps: The use of instance-masked normal maps to inject explicit 3D guidance. This representation offers high geometric density and scale invariance while excluding appearance details, providing orthogonal guidance that avoids feature competition. The accompanying 3D-free data curation strategy enables automated dataset construction using only 2D models.
3. Soft Spatial-Aligned Injection & Minimal Fine-Tuning: A novel injection mechanism and fine-tuning paradigm that maximizes the preservation of pre-trained priors. This approach provides inherent fault tolerance against imperfections in current 3D generation models and achieves rapid adaptation on few-shot data.

Experimental Results

Extensive experiments demonstrate that Real2SAM2Real outperforms state-of-the-art baselines (e.g., ReCamMaster, ViewCrafter, TrajectoryCrafter, WorldForge) in both camera control and entity manipulation.

• Camera Control: On a test set of 100 in-the-wild images with large viewpoint shifts, the proposed method maintains faithful appearance and spatiotemporal coherence. Baseline methods exhibit cascading artifacts and structural breakdowns under similar conditions. Quantitative metrics (ATE, RPE-T, RPE-R) show superior camera fidelity.
• Entity Motion: In scenarios involving randomized object translation and rotation, Real2SAM2Real achieves significantly lower Rotation and Translation errors compared to baselines. It successfully handles complex occlusions and simultaneous multi-entity manipulation without structural degradation.
• Ablation Studies:
  • Removing the Soft Spatial-Aligned Injection (reverting to channel concatenation) leads to appearance distortion and color bleeding, confirming the necessity of decoupling geometry from appearance.
  • Removing Instance-Level Perturbation results in a train-inference discrepancy where the model rigidly adheres to coarse boundaries, causing visual degradation and temporal flickering.
  • The full model achieves the highest scores in VBench metrics for consistency, motion smoothness, and aesthetic quality.

Significance and Claims

The paper claims that Real2SAM2Real fundamentally addresses the limitations of current video generation by decoupling geometry from appearance. By utilizing a generative 3D cache as complementary context, the framework:

• Eradicates Artifacts: It eliminates perspective ambiguities caused by structural holes and erroneous facades, as well as misleading cues from non-Lambertian surfaces (reflections, refractions) that plague traditional "warp-and-inpaint" pipelines.
• Enables Precise Control: It allows for precise, decoupled control over camera trajectories and multi-entity motions, even under large camera shifts and severe occlusions.
• Preserves Priors: Through minimally invasive fine-tuning and soft injection, it retains the powerful zero-shot generalization and photorealistic rendering capabilities of the foundational VDM.
• Democratizes 3D Control: The 3D-free data curation and few-shot adaptation make the framework accessible without requiring massive paired 3D datasets.

The authors position this work as a step toward "taming" VDMs for production-level controllability by providing the necessary 3D-aware guidance that 2D priors alone cannot supply.