💻 computer science

ClipGStream: Clip-Stream Gaussian Splatting for Any Length and Any Motion Multi-View Dynamic Scene Reconstruction

ClipGStream introduces a hybrid reconstruction framework that divides long multi-view dynamic sequences into short clips with independent spatio-temporal fields and inherited anchors, achieving state-of-the-art, flicker-free 3D reconstruction with high temporal coherence and reduced memory overhead.

Original authors: Jie Liang, Jiahao Wu, Chao Wang, Jiayu Yang, Xiaoyun Zheng, Kaiqiang Xiong, Zhanke Wang, Jinbo Yan, Feng Gao, Ronggang Wang

Published 2026-04-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jie Liang, Jiahao Wu, Chao Wang, Jiayu Yang, Xiaoyun Zheng, Kaiqiang Xiong, Zhanke Wang, Jinbo Yan, Feng Gao, Ronggang Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to create a hyper-realistic, 3D movie of a basketball game where the camera can fly around the players, zoom in on a dribble, or watch from the ceiling. This is called Dynamic 3D Scene Reconstruction.

The problem is that these videos are huge. A single game might have thousands of frames. If you try to build the 3D world frame-by-frame, the computer gets confused and the video starts to jitter or "glitch." If you try to build it all at once, the computer runs out of memory and crashes.

The paper introduces ClipGStream, a new way to build these 3D worlds that solves both problems. Here is how it works, using some simple analogies.

The Problem: The Two Bad Options

Before ClipGStream, researchers had two main ways to do this, and both had flaws:

The "Frame-by-Frame" Approach (Frame-Stream):
- Analogy: Imagine a painter who paints one frame of a movie, then throws away their palette and starts a completely new painting for the next frame.
- Result: They can paint a very long movie (scalable), but because they don't remember the previous frame, the characters might jitter, shake, or look like they are vibrating. It's unstable.
The "Whole Clip" Approach (Clip):
- Analogy: Imagine a painter who tries to paint a whole 30-second scene on one giant canvas at once.
- Result: The painting is very smooth and consistent, but the canvas is so heavy the painter can't carry it. They can only paint short scenes before their brain (memory) explodes. They can't paint a whole movie.

The Solution: ClipGStream (The "Relay Race" Strategy)

ClipGStream combines the best of both worlds. It treats the video like a relay race.

1. The Reference Clip (The Starter)

First, the system picks the first short chunk of the video (let's say 30 seconds) and calls it the Reference Clip.

It builds a perfect, stable 3D model of this first chunk.
Think of this as building the foundation and the skeleton of a house. Once the foundation is poured and the frame is up, it stays there. It doesn't move.

2. The Source Clips (The Runners)

Now, the video moves to the next chunk (the next 30 seconds). Instead of starting from scratch, the system inherits the foundation and skeleton from the first chunk.

The Magic Trick (Inheritance): The system says, "Okay, the floor and the walls (static objects) are already built. We don't need to rebuild them." This keeps the background stable and prevents flickering.
The Magic Trick (Residuals): But what if a player runs into a new spot or a ball flies somewhere new? The system adds Residual Anchors.
- Analogy: Imagine the foundation has "hooks" on it. If a new piece of furniture needs to go in the room, you just hang a new hook there. You don't rebuild the whole wall. This allows the system to handle fast, large movements without getting confused.

3. The "Clip-Stream" Flow

The system does this for the whole video:

Clip 1: Build the house.
Clip 2: Keep the house, add new furniture/hooks for new movements.
Clip 3: Keep the house, add more hooks for the next set of moves.
Clip 100: Keep the house, keep adding hooks.

Because it never rebuilds the "house" (the static parts), it stays smooth and stable. Because it only adds small "hooks" (residuals) for new movements, it doesn't need a massive amount of memory.

Why is this a Big Deal?

No More Flickering: In old methods, when the video switched from one short clip to another, the 3D model would "snap" or flicker. ClipGStream keeps the model continuous, like a smooth video stream.
Handles Long Videos: It can process a 1,400-frame video (like a full basketball game) without crashing the computer.
Handles Fast Motion: Whether a player is standing still or sprinting across the court, the system adapts by adding just the right amount of "new hooks" to capture the movement.

The Bottom Line

ClipGStream is like a smart construction crew. Instead of tearing down a building to rebuild it every day (Frame-Stream) or trying to build a skyscraper in one day (Clip), they build a solid base once and then simply add new rooms and decorations as the story progresses. This allows them to build a massive, stable, and high-quality 3D movie of any length, no matter how crazy the action gets.

1. Problem Statement

Dynamic 3D scene reconstruction is critical for immersive media (VR, MR, XR) but faces significant challenges when dealing with long-duration sequences and large-scale motions. Existing methods generally fall into two categories, both of which have limitations:

Frame-Stream Methods: Optimize scenes frame-by-frame. While scalable to long sequences, they suffer from accumulated errors and inter-frame jitter, leading to poor temporal stability.
Clip Methods: Jointly optimize short temporal clips (e.g., 300 frames) to ensure local consistency. However, they incur high memory/computation costs, struggle with long sequences (requiring re-initialization for new clips), and often produce flickering artifacts at clip boundaries due to inconsistent scene representations.

There is a lack of a unified framework that can handle arbitrarily long videos with complex, large-scale motions while maintaining high temporal coherence and reconstruction fidelity.

2. Methodology: ClipGStream

The authors propose ClipGStream, a hybrid framework that divides a long video into a Reference Clip and multiple Source Clips. It employs a "Clip-Stream" paradigm, combining the scalability of stream-based training with the stability of clip-based optimization.

The core of the method consists of two complementary strategies:

A. Intra-clip Training Strategy (Local Optimization)

This strategy handles the modeling of motion within a specific clip.

Feature Decomposition: For the Reference Clip, the scene is modeled by decoupling static features ( $f_s$ $f_{s}$ ) and dynamic features ( $f_d$ $f_{d}$ ).
- Static Features: Represent background and stationary structures.
- Dynamic Features: Captured via a Spatio-Temporal Field (STF) (implemented as a 4D hash grid + MLP) to model local motion and deformations.
Residual Anchor Compensation (RAC): For subsequent Source Clips, the method does not re-optimize the entire scene from scratch. Instead, it inherits the Reference Clip's anchors and adds residual anchors.
- These residual anchors are selected based on a geometry-aware deduplication process. If a candidate anchor from the new clip is far from the existing "spherical coverage field" of the Reference Clip (determined by Signed Distance Function, SDF), it is added as a residual anchor.
- This allows the model to capture newly introduced structures or large displacements without redundant anchor growth.
Independent STFs: Each clip utilizes its own independent STF to model local motion variations, preventing the "forgetting" or overwriting of dynamics that occurs when sharing a single field across long sequences.

B. Inter-clip Inheritance Strategy (Global Consistency)

This strategy ensures temporal stability across different clips.

Static Inheritance: The anchors ( $A_0$ ) and their static features ( $f_{s,0}$ ) from the Reference Clip are inherited by all Source Clips and kept frozen during optimization. This provides a stable geometric foundation for static regions, preventing the flickering caused by re-optimizing static backgrounds.
Decoder Inheritance: The decoder ( $d$ ) trained on the Reference Clip is reused and frozen for all Source Clips. This ensures that the mapping from features to Gaussian attributes (color, opacity, scale) remains consistent across the entire video, significantly improving rendering quality in dynamic regions.

3. Key Contributions

First Clip-Stream Framework: Introduces a novel hybrid paradigm that simultaneously addresses large-scale motion and long-sequence modeling in multi-view dynamic scenes, overcoming the trade-offs between Frame-Stream and Clip methods.
Two-Stage Training Strategy:
- Intra-clip: Uses residual initialization with independent STFs to robustly model complex/fast motions.
- Inter-clip: Freezes shared components (anchors, static features, decoder) to enforce temporal consistency and enable stable reconstruction of arbitrarily long videos.
Geometry-Aware Residual Anchors: A deduplication mechanism that efficiently adds only necessary anchors for new structures, balancing coverage and memory efficiency.
State-of-the-Art Performance: Demonstrates superior reconstruction quality and efficiency across diverse benchmarks.

4. Experimental Results

The method was evaluated on three datasets: Long 360 (1,400 frames, high-speed basketball), VRU (real-world basketball), and N3DV (fine-scale motion, including a 1,200-frame "flame salmon" sequence).

Quantitative Performance:
- On the Long 360 dataset, ClipGStream achieved the highest PSNR (24.54) and lowest LPIPS (0.146), outperforming Frame-Stream methods (e.g., 3DGStream) and Clip methods (e.g., 4DGaussian).
- On the N3DV dataset, it achieved a PSNR of 32.53, surpassing previous SOTA methods like LocalDyGS (32.28) and SpaceTimeGS (32.05), while maintaining a low memory footprint (98MB) and fast training time (0.5h).
- On the Flame Salmon (1,200 frames) sequence, it achieved a PSNR of 29.40, significantly outperforming 4DGaussian (28.89).
Qualitative Results:
- Flicker Suppression: Cross-clip residual heatmaps show that ClipGStream effectively suppresses flickering in static regions, a common failure point for other methods.
- Detail Preservation: The method renders sharper details in both dynamic regions (e.g., athletes, flames) and static backgrounds compared to competitors.
Ablation Studies:
- Removing Decoder Inheritance led to noticeable blurriness.
- Removing Residual Anchor Compensation caused flickering in dynamic areas and poor handling of large motions.
- Removing Anchor Inheritance resulted in severe flickering across clip boundaries.

5. Significance

ClipGStream represents a significant advancement in dynamic 3D reconstruction by solving the "long-sequence" and "large-motion" bottleneck.

Scalability: It enables the reconstruction of videos of any length without the memory explosion or error accumulation seen in previous approaches.
Robustness: It handles arbitrary motion scales, from subtle deformations to rapid, large-scale movements, making it suitable for real-world applications like sports broadcasting and immersive VR/AR.
Efficiency: By freezing shared components and using a hybrid training scheme, it achieves high-quality results with reduced training time and memory overhead compared to joint optimization methods.

The code and project page are publicly available, facilitating further research in scalable dynamic scene reconstruction.