Generative View Stitching

1. Problem Statement

Current autoregressive (AR) video diffusion models, while capable of generating stable, long rollouts, suffer from a fundamental limitation: they cannot condition on the future.

• The Issue: In camera-guided video generation with a predefined trajectory (e.g., for cinematography or autonomous driving simulation), AR models generate frames based only on past context. If the camera trajectory requires the camera to move toward a wall or navigate a complex loop, the model generates content unaware of the future constraints.
• Consequence: This leads to collisions with the generated scene (exposure bias). Once a collision occurs, the generated frames become out-of-distribution for the model, causing the autoregressive process to collapse rapidly.
• Existing Alternatives: Previous "diffusion stitching" methods (e.g., CompDiffuser) exist but typically require training a custom model specifically for stitching, which is computationally expensive and not applicable to off-the-shelf video models.

2. Methodology: Generative View Stitching (GVS)

The authors propose Generative View Stitching (GVS), a training-free sampling algorithm that enables camera-guided video generation by respecting the full predefined camera trajectory (past and future) simultaneously.

Core Mechanism

GVS divides the target video sequence into overlapping chunks and generates the entire sequence in parallel rather than sequentially.

1. Compositional Distribution: Instead of generating frame $t$ conditioned only on $t-1$, GVS models the distribution of a target chunk $x_t$ conditioned on its temporal neighbors ($x_{t-1}, x_{t+1}$) and the camera trajectory.
2. Joint Denoising: The method leverages Diffusion Forcing (DF) backbones. DF models allow independent noise levels per token. GVS feeds the target chunk and its noisy neighbors into the model simultaneously. The model denoises the target chunk while treating the neighbors as conditioning signals, then discards the denoised neighbors and updates the global video estimate with the target chunk.
3. Training-Free: Crucially, this works with any off-the-shelf video model trained with Diffusion Forcing, requiring no retraining or architectural changes.

Key Innovations

To overcome the poor temporal consistency observed in naive parallel stitching, the authors introduce two critical components:

• Omni Guidance:

  • Problem: Standard stitching often results in "hazy" transitions or oversmoothing because the conditioning signal from noisy neighbors is weak.
  • Solution: Omni Guidance modifies the score function to explicitly steer the generation toward consistency with both the past and future. It effectively acts as a generalized classifier-free guidance where the "conditioning" is the model's own co-evolving noisy neighbors.
  • Formula: It adjusts the predicted noise $\tilde{\epsilon}_\theta$ by combining the standard prediction with terms that penalize deviation from the trajectory and temporal neighbors, modulated by a guidance scale $\gamma$.

• Cyclic Conditioning (Loop Closing):

  • Problem: While stitching provides a growing receptive field, it does not inherently enforce global consistency (e.g., returning to the exact same visual state in a loop).
  • Solution: The authors introduce Cyclic Conditioning. During the denoising process, target chunks are alternately conditioned on:
    1. Temporal Windows: Neighboring chunks in time.
    2. Spatial Windows: Chunks that are temporally distant but spatially close (e.g., the start and end of a loop).
  • This ensures that information propagates globally, allowing the video to "visually return" to the same place, enabling complex loops like the Impossible Staircase.

• Stochasticity Control:

  • The method utilizes partial stochasticity ($\sigma_k = \eta\sqrt{1-\alpha_{k-1}}$). While maximum stochasticity improves consistency, it causes oversmoothing. Omni Guidance allows the use of partial stochasticity, reducing oversmoothing while maintaining high temporal consistency.

3. Key Contributions

1. First Training-Free Stitching for Video: GVS is the first method to apply diffusion stitching to camera-guided video generation without requiring a custom-trained model, compatible with any Diffusion Forcing backbone.
2. Omni Guidance: A novel guidance technique that enhances temporal consistency by strengthening conditioning on both past and future states, solving the "weak signal" problem in parallel stitching.
3. Loop-Closing Mechanism: A cyclic conditioning strategy that enables the generation of globally consistent, long-horizon videos that close loops, even for complex trajectories like the "Impossible Staircase."
4. Collision-Free Generation: By planning the entire sequence simultaneously, GVS prevents the camera from colliding with generated geometry, a failure mode common in autoregressive approaches.

4. Experimental Results

The authors evaluated GVS against History-Guided Autoregressive (AR) Sampling and StochSync (a stitching method for images/panoramas) on various challenging trajectories (Straight Line, Stairs, Panorama Loops, Impossible Staircase).

• Metrics:

  • F2FC (Frame-to-Frame Consistency): GVS outperformed baselines.
  • LRC (Long-Range Consistency): GVS achieved significantly better loop closure.
  • CA (Collision Avoidance): GVS achieved 0% collision rate across all tested trajectories, whereas AR sampling frequently collided and collapsed.
  • Quality (IQ/AQ): GVS maintained comparable or superior image and aesthetic quality metrics compared to baselines.

• Qualitative Findings:

  • AR Sampling: Collided with walls, failed to generate the "staircase" correctly (often hitting the ceiling), and produced disjointed loops.
  • StochSync: Avoided collisions but achieved this by "shape-shifting" the scene (drastic changes in geometry), resulting in poor temporal consistency.
  • GVS: Generated stable, collision-free videos that faithfully followed the camera trajectory, successfully closed loops, and maintained high temporal coherence.

• Scalability: GVS demonstrated the ability to generate 120+ frame videos and even 1080-frame videos (in the "Indefinite Staircase" experiment) with stable rollouts.

5. Significance and Impact

• Paradigm Shift: GVS challenges the dominance of autoregressive sampling for long video generation. It demonstrates that non-autoregressive, parallel sampling can be superior when future constraints are known.
• Practical Applications: This method is critical for offline applications requiring high-level planning, such as:
  • One-shot Cinematography: Generating long, continuous shots with complex camera moves.
  • Autonomous Driving Simulation: Creating synthetic data where the camera must navigate specific, collision-free paths.
  • World Models: Enabling neural game engines that respect global scene geometry.
• Efficiency: By leveraging existing Diffusion Forcing models, GVS avoids the massive cost of training new architectures, making advanced long-horizon generation accessible to researchers using standard pre-trained models.

Limitations

• Offline Only: Because it requires the full future trajectory, GVS cannot be used for real-time interactive generation or streaming where future camera poses are unknown.
• External Conditioning: Propagating external context frames (e.g., a single reference image) to the entire video remains challenging.
• Wide-Baseline Loops: The method struggles to close loops involving large viewpoint shifts (wide baselines) if the backbone model was not trained on such data.
• Manual Cyclic Windows: The spatial windows for loop closing are currently manually defined per trajectory, though the authors suggest learning this selection is a future direction.