Technical Summary: GeoStream

Problem Statement

Accurate interactive camera control is a fundamental requirement for video-based world models, enabling applications like immersive scene exploration and robotics simulation. However, existing approaches suffer from significant limitations in controllability:

1. Implicit Control Limitations: Most current video world models inject camera pose embeddings or use control layers implicitly. While visually plausible, these methods lack explicit geometric grounding. They learn camera motion as statistical correlations from data, leading to systematic inaccuracies when input trajectories deviate from the training distribution. Specifically, they exhibit scale ambiguity: increasing the magnitude of input camera translation does not result in proportionally larger motion in the generated video. Instead, the model regresses toward typical trajectories rather than following metric-scale requests.
2. Explicit Control Limitations: Methods that introduce explicit geometric conditioning (e.g., using 3D point clouds) improve controllability but are typically non-autoregressive or rely on a static 3D cache built from an initial frame. Once the viewpoint moves beyond the original frustum, this static geometry becomes ineffective, and the geometric signal degenerates into a noisy, partially observed cue.
3. Exposure Bias in Streaming: Existing streaming methods that attempt to accumulate geometry over time introduce drift. Furthermore, when training autoregressive models with self-generated geometric conditioning, a second-order geometric feedback loop arises: errors in generated frames propagate through the depth estimator into the point reprojections, compounding over time. Standard training techniques (like off-policy condition noising) fail to address this specific error distribution.

Methodology: GeoStream

GeoStream is a framework designed for precise, metric-scale camera control in autoregressive streaming video generation. It addresses the above problems through two core components: a self-refreshing 3D cache and a fully on-policy distillation strategy.

1. Self-Refreshing 3D Cache

Unlike prior work that maintains a persistent, fused 3D memory, GeoStream employs a sliding-window cache refreshing mechanism:

• Periodic Rebuilding: The generation sequence is split into non-overlapping temporal windows (chunks). At the end of each window, the 3D cache is discarded and rebuilt from the most recently generated frame.
• Process: The model estimates depth from the latest generated frame, unprojects pixels into 3D space to form a point cloud, and then reprojects these points into the target camera view.
• Conditioning: The resulting point reprojection image serves as an explicit geometric conditioning signal for the synthesis of subsequent frames.
• Rationale: By discarding stale geometry rather than fusing it, the method avoids compounding inconsistencies (e.g., layering artifacts) and ensures the geometric guidance remains temporally aligned with the current content, even under large camera motions.

2. On-Policy Causal Distillation

To train a causal student model that can reliably condition on this self-refreshed cache, GeoStream addresses two forms of exposure bias: standard autoregressive drift and the second-order geometric feedback loop. The training proceeds in two stages:

• Stage 1: Off-Policy Warmup: A bidirectional teacher model is trained on ground-truth frames and geometric conditioning. A causal student is then initialized via teacher-forced autoregressive diffusion training. In this phase, the student sees clean ground-truth context and conditioning, aligning it with the generation distribution but not yet addressing self-generated errors.
• Stage 2: On-Policy DMD Distillation: The student is distilled using Distribution Matching Distillation (DMD). Crucially, during this stage, the geometric conditioning (point reprojections) is rendered from the student's own generated frames rather than ground truth.
  • This creates a "self-rollout" where the student is supervised against the exact joint distribution of context frames and geometric artifacts it will encounter at inference.
  • This approach naturally aligns the training and inference distributions, effectively closing the second-order feedback loop without relying on hand-crafted noise approximations used in off-policy methods.

Key Contributions

1. GeoStream Framework: A streaming autoregressive video generation framework that enables precise metric-scale viewpoint manipulation under large camera motions by coupling causal synthesis with explicit, online-refreshed 3D geometric conditioning.
2. Periodic 3D Cache Refresh: A mechanism that maintains temporally aligned geometric conditioning by rebuilding the 3D cache from recent generated frames at chunk boundaries, eliminating the need for long-term scene memory or offline reconstruction.
3. Fully On-Policy Training Strategy: A novel training recipe involving off-policy warmup followed by on-policy DMD distillation. This strategy mirrors the inference-time self-refresh mechanism, exposing the student to the true self-generated error distribution and eliminating geometry-induced exposure bias.

Experimental Results

The method was evaluated on the RealEstate10K dataset, comparing against state-of-the-art implicit (e.g., MotionCtrl, SEVA) and explicit (e.g., FlexWorld, ViewCrafter, Spatia) baselines.

• Quantitative Performance: GeoStream achieves state-of-the-art performance across visual fidelity metrics (FVD, PSNR, SSIM, LPIPS) and camera trajectory accuracy metrics (Absolute Trajectory Error - ATE, Relative Translation Error - RTE, Relative Rotation Error - RRE).
  • It significantly outperforms baselines in local trajectory metrics (RTE, RRE), demonstrating superior ability to follow metric-scale motions.
  • Even when baselines are re-evaluated with the same strong depth backbone (MoGe-2) used by GeoStream, GeoStream maintains a lead in FVD and local trajectory metrics, proving the gains are not solely due to the depth estimator.
• Scale Sensitivity: Experiments scaling input camera translation magnitudes show that GeoStream maintains consistent error rates across different scales. In contrast, implicit baselines exhibit "regression-to-the-mean," overshooting small motions and undershooting large ones.
• Robustness: GeoStream demonstrates superior robustness to severe trajectory perturbations (zigzag motions) compared to baselines, which tend to oversmooth high-frequency motion.
• Efficiency: Despite the overhead of depth estimation and cache refreshing, GeoStream achieves a significant inference speedup (4.05 fps) compared to bidirectional baselines (0.4–0.77 fps) on an NVIDIA A100, making it suitable for real-time applications.

Significance and Claims

The paper claims that GeoStream represents a strict upgrade over both implicit and existing explicit camera conditioning methods. Its primary significance lies in demonstrating that explicit, online-refreshed geometry is essential for precise metric-scale control in streaming generation.

The authors emphasize that the key insight is the causal alignment of the geometric conditioning: once the generator is causal, the geometric conditioning must also be causal. By rebuilding the cache from the model's own outputs and training via on-policy distillation, GeoStream successfully mitigates the compounding errors inherent in autoregressive geometric feedback loops. The paper concludes that this mechanism could serve as a plug-in module for other streaming pipelines, though validating this generality is left as future work. The method is modestly limited by its memoryless cache design (lacking explicit long-term loop closure) and inference latency, but it substantially advances the state of the art in streaming camera control.