Technical Summary: Preserve, Reveal, Expand: Faithful 4D Video Editing with Region-Aware Conditioning

1. Problem Statement: Evidence-Role Mismatch

Existing 4D-driven video diffusion models excel at generating plausible videos conditioned on 4D scene representations (e.g., camera motion, object trajectories). However, faithful 4D video editing presents a distinct challenge: the model must simultaneously preserve source-observed regions (retaining identity, texture, and temporal details) and synthesize newly visible, disoccluded, or out-of-view content.

The authors identify a critical failure mode termed Evidence-Role Mismatch. In current approaches, a single conditioning signal (typically a coarse rendered 4D proxy) entangles three distinct types of information:

1. Reliable Evidence: Source-backed observations that should be preserved.
2. Unreliable Cues: Rendered hints for disoccluded regions that lack ground-truth support.
3. Unsupported Regions: Areas outside the original field of view requiring extrapolation.

Treating these uniformly forces the diffusion model to implicitly infer evidence reliability and editing roles, leading to preservation drift (source content degrading), ghosting (leakage of invalid evidence in revealed areas), and unstable extrapolation in expanded views.

2. Methodology: The PREX Framework

To address Evidence-Role Mismatch, the paper proposes PREX (Preserve, Reveal, Expand), a region-aware framework that decomposes the target spatiotemporal volume into three distinct roles based on observation support and scene extent.

2.1 Region Decomposition

For each target frame, pixels are categorized into:

• Preserve ($M^P$): Pixels supported by valid source observations after the edit. These must retain the original appearance and temporal behavior.
• Reveal ($M^R$): Unsupported pixels within the original scene extent (e.g., disocclusions caused by object removal or motion). These require scene-consistent completion.
• Expand ($M^E$): Newly visible areas outside the original field of view. These require coherent out-of-view extrapolation.

2.2 Observation-Backed Appearance Conditioning

Instead of using the raw rendered 4D appearance as a direct RGB condition, PREX constructs an observation-backed appearance field ($C_{rgb}$):

• For Preserve regions, it retrieves appearance from nearby source frames using visibility, depth, instance, and view-time consistency checks.
• For Reveal and Expand regions, it treats the pixels as unsupported, providing only weak or low-confidence conditioning to allow the diffusion model to synthesize content.

2.3 Geometric Confidence

A continuous confidence map ($C_{conf}$) is estimated to measure the reliability of the projected 4D support. This map is derived from projection coverage, instance consistency, and local depth variation. High confidence is assigned to stable, geometrically consistent evidence; low confidence is assigned to sparse or ambiguous regions.

2.4 Region-Aware Adapter

PREX employs a Region-Aware Adapter to inject these controls into a frozen video diffusion backbone (Wan2.1-I2V-14B).

• The adapter encodes the observation-backed appearance ($z_{rgb}$), confidence maps, and region masks ($M^R, M^E$) into control tokens.
• These tokens are injected as residual hints into selected layers of the frozen backbone.
• This design allows the model to learn role-specific behaviors (preservation vs. synthesis) without retraining the entire backbone, maintaining strong visual quality while adapting to editing constraints.

2.5 Training via Proxy-Task Curriculum

Since paired data (source video + edited 4D scene + edited video ground truth) is scarce, PREX utilizes a proxy-task curriculum trained on unedited videos. The curriculum simulates the three roles:

1. Preservation: Training the model to retain regions supported by reliable observations.
2. Reveal: Training the model to complete synthetically withheld in-scene areas.
3. Expand: Training the model to extrapolate content beyond the observed view window.

3. PREBench: A Diagnostic Benchmark

The authors introduce PREBench, a diagnostic benchmark designed to evaluate faithful 4D editing with region-aware metrics, moving beyond holistic video quality scores.

• Dataset: Curated from six datasets (DynPose-100K, UVO, PointOdyssey, etc.) with 10,000 training samples and 350 real-world editing test cases (camera-only and joint camera-object).
• Metrics:
  • Preserve: P-LPIPS, P-DISTS, P-TempDrift (measuring source fidelity and temporal stability).
  • Reveal: R-Ghost (measuring leakage of invalid evidence), R-Seam (boundary continuity).
  • Expand: E-Temp (temporal stability), E-Seam, E-Copy (detecting degenerate copying/stretching of boundary textures).
• Validation: Human preference studies confirm that metrics like R-Ghost and E-Copy align well with human judgments of specific artifacts.

4. Experimental Results

Experiments demonstrate that PREX outperforms state-of-the-art baselines (e.g., DaS, GEN3C, VerseCrafter, NeoVerse) in both global quality and region-specific fidelity.

• Global Quality: PREX achieves the highest VBench Overall Score (77.99), indicating competitive perceptual realism and temporal smoothness.
• Region-Aware Fidelity:
  • Preserve: PREX achieves the lowest error in P-LPIPS (0.2137) and P-TempDrift, significantly reducing preservation drift compared to baselines.
  • Reveal: PREX obtains the lowest R-Ghost score (0.1374), effectively suppressing ghost artifacts in disoccluded areas.
  • Expand: PREX shows the best E-Temp (0.0615) and E-Copy (0.7085) scores, indicating more stable and non-degenerate out-of-view synthesis.
• Control Accuracy: PREX achieves the lowest camera rotation and translation errors in both camera-only and joint camera-object settings, demonstrating precise adherence to 4D edits.
• User Study: In a 2AFC study, PREX was preferred over baselines in 82.6% of cases for motion alignment, 81.9% for input context preservation, and 81.8% for perceived visual quality.

5. Significance and Claims

The paper claims that separating evidence reliability from editing role is fundamental to faithful 4D video editing. By explicitly decomposing the problem into Preserve, Reveal, and Expand roles, PREX resolves the Evidence-Role Mismatch that plagues existing unified conditioning approaches.

• Key Contribution: The introduction of a region-aware framework that enables observation-backed preservation while allowing generative synthesis for unsupported regions, all within a single frozen diffusion backbone.
• Diagnostic Value: PREBench provides a granular evaluation tool that localizes specific failure modes (ghosting, drift, copying) rather than relying on aggregate scores, making the limitations of 4D editing methods more interpretable.
• Modesty: The authors acknowledge that PREX depends on the accuracy of upstream 4D reconstruction. Errors in geometry, camera poses, or visibility estimation can propagate to region masks, and the method may struggle with complex dynamic occlusions, non-rigid motion, or transparent objects where reliable 4D evidence is difficult to obtain.

In summary, PREX advances the field by shifting the paradigm from "plausible generation" to "faithful editing," ensuring that source evidence is preserved where valid and synthesized only where necessary.