💻 computer science

RiGS: Rigid-aware 4D Gaussian Splatting from a Single Monocular Video

RiGS is a novel 4D Gaussian Splatting framework that reconstructs dynamic 3D scenes from single monocular videos by decomposing the scene into static, rigid, and transient Gaussian primitives to simultaneously capture long-term smooth transformations and short-term complex deformations, achieving state-of-the-art performance in novel view synthesis.

Original authors: Chenyu Wu, Wanhua Li, Zhu-Tian Chen, Hanspeter Pfister

Published 2026-05-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chenyu Wu, Wanhua Li, Zhu-Tian Chen, Hanspeter Pfister

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to recreate a moving 3D world (like a person dancing or a car driving) using only a single video taken with a regular phone camera. This is a huge puzzle for computers. The paper introduces a new solution called RiGS (Rigid-aware 4D Gaussian Splatting) to solve this puzzle.

Here is how it works, explained through simple analogies:

The Problem: One Size Doesn't Fit All

Previous methods tried to describe every moving part of a video using the same "tool."

Some tools were good at describing slow, smooth movements (like a car driving down a straight road), but they failed when things moved fast or changed shape quickly (like a dog wagging its tail).
Other tools were good at fast, chaotic movements, but they made the video look jittery and unstable over time.

The authors realized that real-world motion is like a two-peak mountain range: one peak for long, smooth movements and another sharp peak for short, fast, complex movements. Trying to use one tool to climb both peaks just doesn't work well.

The Solution: A Three-Tool Kit

RiGS solves this by using three different types of "digital building blocks" (called Gaussian primitives) to build the 3D scene, depending on what is happening in the video:

The Static Blocks (The Background):
Think of these as the walls of a room. They never move. Whether the video is at the beginning or the end, these blocks stay exactly where they are to represent the floor, walls, or sky.
The Rigid Blocks (The Smooth Dancers):
Imagine a rigid robot arm or a solid wooden box. These blocks move together as a single unit. They are perfect for things that slide, rotate, or drive smoothly without changing their shape. They handle the "long-term" smooth motions.
The Transient Blocks (The Fireworks):
These are like fireworks or a swarm of bees. They are designed to appear for a very short time, move in crazy directions, and then disappear. They handle the "short-term" fast, complex, or wiggly motions (like a flapping flag or a shaking dog tail) that the rigid blocks can't handle.

The Magic Trick: Switching Roles

The clever part of RiGS is that these blocks can change roles while the computer is learning.

If a "Rigid Block" (the smooth dancer) tries to model a movement that is too fast or too wiggly, it realizes it's struggling.
The system then says, "Okay, you're not a smooth dancer anymore; you're a firework!" and it transforms into a Transient Block.
This allows the system to automatically decide which tool is best for every single part of the video, ensuring the result is both smooth and detailed.

The "Object" Rule

To make sure the computer doesn't get confused, RiGS uses a special rule called the Object-Wise Dynamic Mask.

Old way: The computer looked at individual pixels. If a dog's tail was wagging but the rest of the dog was still, it might get confused and think the whole dog was half-moving and half-still.
RiGS way: The computer looks at the whole object (the whole dog). If any part of the dog moves, the computer treats the entire dog as a moving object. This keeps the 3D model consistent and prevents the video from looking glitchy or broken.

The Result

By using this mix-and-match approach, RiGS can take a single video and turn it into a high-quality 3D scene that you can look at from any angle.

It captures the smooth, long movements (like a person walking) without blurring them.
It captures the fast, detailed movements (like facial expressions or flapping clothes) without making the video jittery.

In short, RiGS is like a smart construction crew that knows exactly when to use a steady crane, a flexible robot arm, or a swarm of drones to build a perfect 3D movie from a single video file.

Technical Summary: RiGS – Rigid-aware 4D Gaussian Splatting from a Single Monocular Video

Problem Statement

Reconstructing dynamic 3D scenes from a single monocular video is a fundamental challenge in computer vision and graphics, with applications in VR/AR, autonomous driving, and robotics. Existing methods face a critical trade-off when modeling motion across different temporal scales:

Per-frame optimization methods can capture high-frequency motion details but often suffer from poor temporal consistency and unstable geometry.
Rigid-based methods (e.g., those using "as-rigid-as-possible" assumptions) maintain geometric stability and handle large viewpoint variations well but fail to model regions with large deformations or fast, non-rigid motion, leading to incomplete reconstructions.

The authors identify a specific limitation in rigid-based Gaussian representations: they struggle to simultaneously capture long-term smooth transformations and short-term complex deformations. Analysis of rigid-based dynamic Gaussians reveals a bimodal distribution in temporal duration, indicating that rigid representations effectively model stable, long-term motion but fail to represent short-term, fine-grained dynamics.

Methodology: RiGS

The proposed Rigid-aware 4D Gaussian Splatting (RiGS) framework addresses these limitations by decomposing scene motion into three distinct types of Gaussian primitives within a unified optimization framework:

1. Three-Tier Gaussian Representation

Static Gaussians ( $g_s$ ): Represent time-invariant backgrounds that persist throughout the entire video.
Rigid Gaussians ( $g_r$ ): Designed to model long-term, low-frequency, smooth motions. These are parameterized with SE(3) transformations (rotation and translation without deformation) and shared motion bases to ensure coherent movement.
Transient Gaussians ( $g_t$ ): Designed to handle short-term, high-frequency, and complex non-rigid dynamics. These possess fully learnable motion velocity vectors, allowing them to move freely in any direction to capture rapid changes.

2. Object-wise Dynamic Decomposition

To guide the separation of static and dynamic regions, RiGS introduces object-wise dynamic masks. Unlike previous pixel-level approaches that often lack temporal and spatial coherence across entire objects, RiGS aggregates long-range spatiotemporal motion information at the object level.

The method uses SAM2 for object segmentation and computes motion scores based on optical flow uncertainty and Sampson error.
If any part of an object exhibits motion at any time, the entire object is treated as dynamic. This prevents static and dynamic Gaussians from redundantly modeling different portions of the same object, ensuring globally consistent representation.

3. Adaptive Transition and Joint Optimization

Soft Gating and Transition: Rigid Gaussians are equipped with a soft gating function that controls their temporal opacity decay based on duration ( $\beta$ ) and center ( $\gamma$ ). During optimization, rigid Gaussians with short temporal durations are dynamically converted into transient Gaussians. This allows the system to adaptively shift from rigid modeling to flexible modeling as motion complexity increases.
Scene Flow Guidance: To provide dense 3D motion supervision, the method lifts 2D optical flow to 3D scene flow using metric depth and camera parameters. This scene flow guides the motion directions of both rigid and transient Gaussians, addressing the sparsity of supervision signals for dynamic regions.

4. Optimization Strategy

The framework employs a joint optimization strategy involving:

Warm-up Stage: Static Gaussians, rigid Gaussians, and motion bases are initialized to establish stable geometry.
Joint Training: All three Gaussian types are optimized together. The system uses photometric loss, geometric loss (depth and normal consistency), correspondence loss (tracking and flow), and regularization terms to prevent artifacts and overfitting.

Key Contributions

Multi-Scale Motion Modeling: RiGS introduces a unified framework that jointly models static, rigid, and transient Gaussians, enabling the capture of motions across multiple temporal scales (from long-term smooth motion to short-term high-frequency dynamics).
Object-wise Dynamic Masking: The proposal of object-wise dynamic masks for static-dynamic decomposition, which aggregates spatiotemporal information to ensure temporal consistency and prevent inconsistent object representation.
Joint Optimization with Scene Flow: A strategy that guides the learning of motion across scales using dense 3D scene flow supervision and a mechanism for rigid Gaussians to transition into transient Gaussians based on temporal duration.

Experimental Results

The authors evaluated RiGS on several benchmarks, including the Nvidia Dynamic Scenes Dataset, the DyCheck iPhone Dataset, and extreme motion scenes from the DAVIS dataset.

Quantitative Performance: RiGS achieved state-of-the-art performance on novel view synthesis benchmarks. On the Nvidia Dynamic Scenes dataset, it outperformed previous SOTA methods (including NeRF-based and Gaussian-based approaches like MoSca and DynPoint) in PSNR, SSIM, and LPIPS metrics. For instance, on the "Playground" scene, RiGS achieved a PSNR of 24.73 compared to MoSca's 24.25.
Qualitative Performance:
- In scenes with large non-rigid deformations (e.g., the "Umbrella" scene), RiGS successfully modeled the deformation without the artifacts or inconsistent geometry seen in previous methods.
- In complex multi-object scenes (e.g., "Playground"), RiGS preserved fine-grained details (such as facial expressions) that rigid-only or node-interpolated methods failed to reconstruct.
- On extreme motion scenes (e.g., "Judo" and "Dog-agility" from DAVIS), RiGS successfully captured fast-moving regions where other methods failed to recover details.
Efficiency: The method trains a 270-frame video (480×360) in approximately 30 minutes and achieves a rendering speed of around 160 FPS.

Significance and Claims

The paper claims that RiGS represents a significant advancement in dynamic scene reconstruction by explicitly addressing the challenge of modeling motion across multiple temporal scales. By decomposing the scene into static, rigid, and transient components, the method overcomes the limitations of purely rigid or purely per-frame approaches.

The authors assert that their object-wise dynamic mask provides a more reliable separation of static and dynamic regions, while the transition mechanism between rigid and transient Gaussians, combined with scene-flow-guided optimization, provides dense and coherent 3D motion supervision. The extensive experiments demonstrate that this approach leads to superior reconstruction quality and temporal consistency, paving the way for more robust dynamic scene modeling in real-world applications.