💻 computer science

RoDyGS: Robust Dynamic Gaussian Splatting for Casual Videos

This paper introduces RoDyGS, a method that reconstructs dynamic 3D scenes from casual monocular videos by explicitly separating static and dynamic elements with spatiotemporal regularization, while also proposing a new comprehensive benchmark, Kubric-MRig, to evaluate such pose-free dynamic novel view synthesis approaches.

Original authors: Junmyeong Lee, Hoseung Choi, Yoonwoo Jeong, Minsu Cho

Published 2026-05-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Junmyeong Lee, Hoseung Choi, Yoonwoo Jeong, Minsu Cho

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are holding your phone and recording a casual video of a busy park. You see people walking, dogs running, and trees swaying in the wind. Now, imagine trying to turn that flat, 2D video into a 3D world where you can walk around the trees and watch the dogs from a different angle. This is incredibly hard because your video only shows one perspective, and the camera itself is moving, making it difficult to tell what is moving because the camera moved versus what is moving on its own.

This paper introduces RoDyGS, a new tool designed to solve this puzzle. Think of RoDyGS as a "digital time-traveling sculptor" that takes your shaky, handheld video and rebuilds the entire 3D scene, including how objects move over time.

Here is how it works, broken down into simple concepts:

1. The "Static vs. Dynamic" Sorting Hat

The biggest problem with these videos is confusion. Is that tree moving because the wind blew it, or because you walked past it?

The Analogy: Imagine a chaotic party where some guests are standing still (the walls, the floor) and others are dancing wildly (people, cars). If you try to describe the whole room at once, it's a mess.
The Solution: RoDyGS puts on a "sorting hat." It immediately separates the scene into two groups: Static (the background that doesn't move) and Dynamic (the things that do). By treating them separately, the system stops getting confused about whether the camera is moving or the object is moving.

2. The "Ghost-Busting" Regularization

When you try to reconstruct a moving object from a single video, parts of it often disappear behind other things (occlusion). If a person walks behind a car, the system loses track of them for a moment. Without help, the 3D model might glitch, turning the person into a blob or a ghostly smear.

The Analogy: Imagine trying to guess the shape of a toy car while someone keeps covering it with a blanket. If you just guess randomly, you might think the car is melting.
The Solution: RoDyGS uses "rules of physics" (called regularization) to keep the model honest.
- Distance-Preserving: It reminds the system, "Hey, the distance between the car's wheels shouldn't change just because you can't see them for a second." It keeps the object rigid.
- Surface Smoothing: It ensures the surface of the object stays smooth, like a real ball, rather than looking like a jagged, pixelated rock.
- Steady Motion: It assumes that objects don't teleport or jitter wildly between frames. If a ball is rolling, it keeps rolling smoothly, not vibrating in place.

3. The "Self-Correcting Camera"

Usually, to build a 3D model, you need to know exactly where the camera was at every second. But in casual videos, you don't have that data.

The Analogy: It's like trying to draw a map of a city while walking through it blindfolded, only knowing what you see in front of you.
The Solution: RoDyGS acts like a detective. It looks at the static parts of the scene (the ground, the buildings) to figure out where the camera must have been. It constantly corrects its own guess about the camera's path while simultaneously building the 3D model.

4. A New "Training Gym" (Kubric-MRig)

The authors realized that existing tests for these tools were too easy or unrealistic. They were like testing a race car on a flat, empty parking lot.

The Analogy: They built a new "obstacle course" called Kubric-MRig. This is a synthetic world generated by a computer where they can control everything. They created scenes with wild camera movements, objects flying around, and complex backgrounds, all with a "perfect answer key" (ground truth) so they could prove their tool actually works.

The Results

When they tested RoDyGS on this new obstacle course and on real-world iPhone videos, it performed better than previous methods.

The Outcome: It creates clearer, sharper 3D movies. If you watch a video of a spinning fan, RoDyGS can reconstruct the fan blades so they look solid and real, whereas older methods might make them look like a blurry smear or a ghost.

In summary: RoDyGS is a smart system that separates moving things from stationary ones, uses strict rules to keep moving objects from turning into digital ghosts, and figures out the camera's path on its own. This allows it to turn a simple, shaky phone video into a high-quality, 3D, time-traveling experience.

Technical Summary: RoDyGS: Robust Dynamic Gaussian Splatting for Casual Videos

1. Problem Statement

Reconstructing 4D scenes (3D geometry + time) from casually captured monocular videos is a significant challenge in computer vision. While recent advances in Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have improved static novel view synthesis, dynamic reconstruction from single-camera, uncalibrated videos remains difficult due to two primary factors:

Camera-Object Motion Ambiguity: Jointly estimating camera trajectories and dynamic scene geometry from a single view is inherently ambiguous.
Occlusions and Spatiotemporal Consistency: Dynamic objects are frequently occluded, leading to missing observations that disrupt the consistency of scene elements over time.

Existing methods often rely on specialized capture setups (multi-view, depth sensors), pre-calibrated poses, or strict scene constraints (e.g., forward-facing cameras, small motions), limiting their applicability to general "in-the-wild" scenarios. Furthermore, current benchmarks often lack ground truth camera poses or concurrent multi-view captures, making rigorous evaluation of pose-free dynamic methods difficult.

2. Methodology: RoDyGS

The authors propose RoDyGS (Robust Dynamic Gaussian Splatting), an end-to-end framework that reconstructs dynamic scene representations directly from casual monocular videos without external sensors or manual calibration. The method optimizes dynamic Gaussian splatting and camera motion jointly.

Core Components

Static-Dynamic Part Separation:
The scene is explicitly partitioned into two sets of Gaussians: static ( $G_S$ ) and dynamic ( $G_D$ ).
- Static Elements: Optimized jointly with camera parameters to ensure temporally coherent background representation.
- Dynamic Elements: Optimized independently for motion constraints.
- Benefit: This separation disentangles camera motion from dynamic object motion, significantly reducing ambiguity and improving optimization stability.
Initialization:
- Camera Poses & Geometry: Initialized using MASt3R [36], a data-driven prior model, to provide robust initial camera poses and point clouds.
- Motion Masks: Dynamic regions are identified using the Track Anything Model (TAM) [65] to separate moving objects from the background.
- Depth: Monocular depth maps are estimated using DepthAnything [66] for geometric supervision.
Spatiotemporal Regularization:
To enforce physically plausible geometry and natural motion, RoDyGS introduces specific regularization terms for dynamic Gaussians:
1. Distance-Preserving Regularization ( $L_{dp}$ ): Enforces temporal consistency by maintaining the relative distances between a Gaussian and its $K$ nearest neighbors across time steps. This preserves the rigidity of objects even during occlusions.
2. Surface Smoothing Regularization ( $L_{surf}$ ): Ensures local surface smoothness by penalizing deviations of a Gaussian from the mean position of its neighbors, preventing local perturbations.
3. Steady Motion Regularization ( $L_{sm}$ ): Based on the assumption that real-world motion is consistent over short intervals, this term regularizes the motion basis functions to prevent drift.
4. Weighted Motion Regularization: To avoid over-smoothing complex motions, a weighted scheme is applied to the motion basis elements, allowing explicit control over the regularization strength for different motion frequencies.
Optimization Objective:
The framework jointly optimizes Gaussian attributes, motion basis parameters, and camera poses using a composite loss function:
- Photometric Loss ( $L_{recon}$ ): L1 and SSIM losses between rendered and target images.
- Depth Loss ( $L_{depth}$ ): Pearson depth loss to maximize linear correlation between rendered and estimated monocular depth (handling scale ambiguity).
- Geometric Loss ( $L_{geo}$ ): Combination of $L_{dp}$ and $L_{surf}$ .
- Motion Loss ( $L_{sm}$ ): Steady motion regularization.

3. Key Contributions

RoDyGS Framework: A robust reconstruction method for casual monocular videos that achieves high-fidelity rendering through explicit part separation and spatiotemporal regularization. It outperforms previous pose-free dynamic novel view synthesis approaches.
Kubric-MRig Benchmark: A new, challenging dataset designed to evaluate both pose estimation and rendering quality in dynamic scenes. Unlike previous benchmarks, Kubric-MRig features:
- Simultaneous multi-view captures (ground truth).
- Extensive camera and object motion.
- Ground truth camera poses.
- Background elements (often missing in other dynamic datasets).
Comprehensive Evaluation: Demonstrated state-of-the-art performance on the Kubric-MRig dataset and competitive results on the iPhone dataset [15] and Tanks and Temples dataset [30], matching the rendering quality of pose-free static methods while handling dynamics.

4. Experimental Results

Dynamic Novel View Synthesis: On the Kubric-MRig dataset, RoDyGS significantly outperforms prior pose-free methods (e.g., RoDynRF [39], SoM [57]) in PSNR, SSIM, and LPIPS. It also achieves competitive results against methods trained with ground truth camera poses.
Static Novel View Synthesis: When evaluated on static scenes (Tanks and Temples), RoDyGS achieves rendering quality comparable to CF-3DGS [13] and significantly better than other baselines, while maintaining competitive camera pose estimation accuracy.
Ablation Studies: Experiments confirm that both part separation and the proposed regularization terms are critical. Removing part separation leads to severe inconsistencies and shape distortions, while the regularization terms sharpen object boundaries and maintain structural consistency.
Robustness: RoDyGS maintains rigid object geometry even under large camera and object motions, whereas baselines relying on long-term point tracking often fail due to inaccurate tracks in complex dynamics.

5. Significance and Limitations

Significance:
The paper claims that RoDyGS addresses the critical gap in reconstructing dynamic 4D scenes from casual, single-camera videos without relying on specialized hardware or pre-calibrated poses. By introducing a comprehensive benchmark (Kubric-MRig) that includes ground truth and diverse motion, the authors provide a rigorous standard for evaluating future dynamic reconstruction methods. The method's ability to separate static and dynamic elements allows for more stable optimization in ambiguous scenarios.

Limitations:
The authors acknowledge several limitations:

Occlusion: Like other 4D reconstruction methods, RoDyGS can only reconstruct observed parts of the scene. Completely occluded regions or long-term occlusions cannot be recovered without generative priors (e.g., diffusion models).
Rigidity Assumption: The regularization terms assume dynamic elements do not exhibit highly complex, non-rigid deformations (e.g., human bodies or loose clothing). Generalizing to such scenarios would require parametric models (e.g., SMPL) or stronger data-driven priors.
Initialization Dependency: The method relies on MASt3R for initial camera poses; inaccurate initializations can diminish rendering quality, though the joint optimization helps refine them.

In conclusion, RoDyGS represents a step forward in making 4D reconstruction accessible for everyday, casually captured videos, offering a robust solution that balances geometric consistency with motion fidelity.