Technical Summary: Efficient Camera-Controlled Video Generation of Static Scenes via Sparse Diffusion and 3D Rendering

Problem Statement

Modern video generative models based on diffusion (e.g., Sora, Wan) achieve high visual fidelity but suffer from extreme computational inefficiency. Generating even short clips often requires minutes of GPU time because these models iteratively denoise every single frame. This bottleneck prevents real-time deployment in applications requiring interactive or embodied AI, such as VR/AR and dynamic content creation. While recent works have attempted to improve efficiency via latent space architectures or distillation, they still rely on neural networks to generate every frame, failing to exploit the inherent redundancy in video signals where many frames depict the same underlying 3D scene under varying viewpoints.

Existing camera-controlled video generation methods either encode camera poses directly into generative models or use 3D priors as internal constraints, yet they still produce every frame via diffusion. Consequently, they have not fundamentally addressed the efficiency barrier.

Methodology: SRENDER

The authors propose SRENDER, a framework that generates full videos by synthesizing a sparse set of keyframes via diffusion and then synthesizing the dense video through 3D reconstruction and rendering. The pipeline consists of four main stages:

1. Adaptive Keyframe Selection

Instead of generating a fixed number of frames, SRENDER employs a Keyframe Density Predictor (a transformer-based model) to determine the optimal number of keyframes for a given camera trajectory and scene appearance.

• Input: A sequence of camera poses and a global feature token extracted from the input image (via DINOv2).
• Mechanism: The model predicts the required number of keyframes (ranging from 4 to 35 for 20-second videos). Simple, smooth trajectories require fewer keyframes, while complex motions with large viewpoint changes require denser sampling.
• Training: Ground truth densities are derived automatically using a coverage-based algorithm (Algorithm 1) that iteratively selects frames until the projected point cloud coverage falls below a threshold.

2. Sparse Keyframe Diffusion

A History-Guided Video Diffusion model generates the selected sparse keyframes.

• Architecture: The model utilizes a diffusion-forcing architecture where each frame is assigned an independent noise level, allowing for flexible conditioning.
• Training Strategy: To handle the instability of large viewpoint jumps at low frame rates, the model is first trained on high-frame-rate videos and then progressively subsampled to match the sparse keyframe spacing used during inference.
• Inference: For trajectories requiring more than 8 keyframes (the context window size), a two-stage scheme is used: first generating 8 keyframes spanning the trajectory, then generating subsequent keyframes conditioned on the nearest previously generated ones.

3. 3D Reconstruction

The generated sparse keyframes are fed into a deterministic, feed-forward 3D Reconstruction Model (specifically AnySplat).

• Process: AnySplat predicts a 3D Gaussian Splatting (3DGS) representation directly from the unposed keyframes.
• Alignment: Since the predicted poses from AnySplat may differ from the input trajectory, a least-squares affine transformation is computed to align the 3D scene with the target camera path.

4. Temporal Chunking and Rendering

To address drift in long trajectories (e.g., >10 seconds) where global 3D consistency is difficult to maintain:

• Chunking: The video is divided into temporal chunks (e.g., 10 seconds). Each chunk undergoes independent 3D reconstruction.
• Alignment: Adjacent chunks share a keyframe, and an affine transformation is estimated to align the coordinate frames of neighboring chunks, ensuring seamless transitions.
• Rendering: The final dense video is rendered at high frame rates using standard 3DGS rasterization, which is significantly faster than diffusion-based synthesis.

Key Contributions

1. Paradigm Shift: The paper challenges the paradigm of generating every frame with neural networks. Instead, it leverages scene redundancy by generating only sparse keyframes and filling the gaps via deterministic 3D rendering.
2. Adaptive Computation: The introduction of a keyframe density predictor allows the system to allocate computational resources dynamically based on motion complexity, avoiding redundant generation for simple scenes.
3. Integration of 3D Priors: Unlike previous works that use 3D priors only for internal constraints, SRENDER uses 3D reconstruction as the primary mechanism for video synthesis, enforcing geometric consistency and enabling real-time rendering.
4. Temporal Chunking: A strategy to handle long-horizon generation by breaking the problem into consistent local 3D reconstructions, mitigating the drift issues common in long-range diffusion models.

Experimental Results

The method was evaluated on RealEstate10K (RE10K) and DL3DV datasets, comparing against the History-Guided (HG) diffusion baseline and the Voyager model.

• Efficiency: SRENDER achieves a 43× speed-up over the HG baseline on DL3DV (generating 20-second, 30-fps videos in ~16 seconds vs. ~697 seconds) and a 23.7× speed-up on RE10K. It achieves real-time performance (37 fps generation rate) on DL3DV.
• Quality: Quantitative metrics (FID and FVD) show that SRENDER outperforms the HG baseline in both image and video quality. It also surpasses Voyager in FID and FVD on the 5 fps test set.
• Comparison with 2D Interpolation: SRENDER significantly outperforms 2D interpolation methods (FILM, RIFE) in FVD, avoiding morphing artifacts and maintaining geometric stability, while also being faster due to the efficiency of 3DGS rendering.
• Ablation: Removing temporal chunking leads to degraded quality (higher FVD) on long sequences, confirming the necessity of the chunking strategy for consistency.

Significance and Claims

The paper claims that explicit 3D reasoning and adaptive sparse generation can dramatically reduce the computational cost of video synthesis without sacrificing fidelity. The authors position SRENDER not as a replacement for all video generation tasks, but as a highly effective solution for camera-controlled generation of static scenes.

The significance lies in providing a practical path toward real-time, controllable video synthesis. By decoupling the generation of scene appearance (via sparse diffusion) from the synthesis of temporal density (via 3D rendering), the method overcomes the efficiency barriers that currently limit the deployment of generative video in interactive applications like embodied AI and VR/AR. The authors note that while their current method is limited to static scenes, the core principles are designed to be transferable to dynamic 4D reconstruction as those models mature.