ODE-GS: Latent ODEs for Dynamic Scene Extrapolation with 3D Gaussian Splatting

Technical Summary: ODE-GS: Latent ODEs for Dynamic Scene Extrapolation with 3D Gaussian Splatting

ODE-GS is a novel framework designed to solve the problem of dynamic scene extrapolation—the ability to predict and render future 3D scene states beyond the observed temporal window. While existing 3D Gaussian Splatting (3DGS) methods excel at temporal interpolation (filling gaps between known frames), they typically fail at extrapolation because they rely on explicit timestamp conditioning, which leads to out-of-distribution (OOD) failures when predicting future times.

1. The Problem: Interpolation vs. Extrapolation

Most current dynamic scene reconstruction methods (e.g., 4D-GS, Deformable-GS) use a deformation network conditioned on a specific timestamp $t$. During training, $t$ is bounded by $[t_{min}, t_{max}]$. When asked to render a frame at $t > t_{max}$, the model encounters a timestamp it has never seen, causing the reconstruction to degrade or collapse. The challenge is to move from a "timestamp-conditioned" paradigm to a "motion-conditioned" paradigm that can forecast continuous, physically plausible trajectories.

2. Methodology

The authors propose a decoupled, two-stage pipeline that separates scene reconstruction from temporal forecasting:

• Stage 1: Interpolation Model (Data Generation):
  The authors first train a standard canonical-plus-deformation 3DGS model. This model learns to reconstruct the scene within the observed window. Once trained, this model is frozen and used as a "ground truth" generator to produce dense, continuous trajectories of Gaussian parameters (position, rotation, scale) for the observed period.
• Stage 2: Transformer-Latent ODE (Forecasting):
  This is the core innovation. Instead of predicting pixels or explicit timestamps, the model treats extrapolation as a sequence-to-sequence problem in a continuous latent space:
  • Encoder: A Transformer processes a sequence of past Gaussian trajectories to extract a latent state $z(t_0)$ that summarizes the motion dynamics.
  • Latent ODE: A Neural Ordinary Differential Equation (ODE) evolves this latent state forward in time ($\dot{z} = f_\theta(z(t))$). This enforces a physical prior of spatio-temporal smoothness, as the ODE naturally models continuous motion.
  • Decoder: A network maps the evolved latent states back into future Gaussian parameters.
• Dynamic Trajectory Sampling: To ensure robustness, the model is trained on varying forecasting horizons (short-term and long-term) by sampling different "prefix" (input) and "suffix" (target) segments from the generated trajectories.
• Regularization: To prevent unphysical oscillations, the authors apply two regularizers:
  1. Latent Regularization: Penalizes high-frequency changes in the ODE velocity field.
  2. Trajectory Regularization: Penalizes acceleration in the decoded 3D Gaussian positions.
  3. Adaptive Weighting: A mechanism that increases the strength of these regularizers as the model converges, guiding it toward smoother solutions.

3. Key Contributions

• Continuous-Time Latent Dynamics: By modeling motion via Neural ODEs rather than discrete timestamps, the model avoids OOD errors and enables rendering at any arbitrary future time.
• Decoupled Training Strategy: Separating reconstruction (interpolation) from forecasting (extrapolation) allows the model to learn from high-quality trajectories without the instability of end-to-end training.
• Physical Inductive Biases: The integration of latent and trajectory smoothness regularizers ensures that predicted motions are physically plausible.

4. Results and Performance

The model was evaluated on three major benchmarks: D-NeRF, NVFi, and HyperNeRF.

• State-of-the-Art (SOTA): ODE-GS significantly outperformed existing methods (like 4D-GS and GaussianPrediction). On the NVFi dataset, it improved metrics by an average of 21.4% PSNR, 7.4% SSIM, and 30.5% LPIPS compared to leading baselines.
• Robustness: In complex scenarios (e.g., the "factory" or "darkroom" scenes in NVFi), the model showed much lower perceptual error (LPIPS) than baselines, proving its ability to handle occlusions and cluttered environments.
• Stability: Qualitative results (residual maps) demonstrate that ODE-GS maintains much lower error in object position and shape compared to methods that suffer from "jitter" or "drift" during extrapolation.

5. Significance

ODE-GS represents a significant step toward intelligent 3D scene understanding. By bridging 3D Gaussian Splatting with Neural ODEs, the paper moves the field from "reconstructing what was seen" to "predicting what will happen." This capability is foundational for high-stakes applications such as autonomous driving, robotics, and augmented reality, where a system must anticipate the future movement of dynamic objects to navigate or interact safely.