💻 computer science

MVFusion-GS: Motion-Variance Guided Temporal Attention for High-Quality Dynamic Gaussian Splatting

MVFusion-GS enhances dynamic 3D Gaussian Splatting by introducing a motion-variance guided refinement mechanism for dynamic-static separation and a MotionFormer temporal attention module to model local motion dependencies, thereby achieving state-of-the-art performance in both dynamic scene reconstruction and distractor-free scenarios.

Original authors: Jianwei Hu, Tingxuan Huang, Hengyu Zhou, Ningna Wang, Xiaohu Guo Jinshan Lai, Bin Wang

Published 2026-07-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jianwei Hu, Tingxuan Huang, Hengyu Zhou, Ningna Wang, Xiaohu Guo Jinshan Lai, Bin Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to take a perfect, crystal-clear photo of a beautiful garden (the static background) while a group of kids is running around playing tag (the dynamic foreground).

In the world of 3D computer graphics, there's a popular technique called 3D Gaussian Splatting. Think of it like building a 3D scene out of millions of tiny, fuzzy, colored clouds (Gaussians). When you want to see the scene from a new angle, the computer blends these clouds together to create a new image.

The Problem: The "Ghost" Effect

The paper explains that when we try to make this work for moving scenes, the computer gets confused. It tries to separate the "moving kids" from the "still garden."

However, older methods (like the one called DeGauss) are a bit clumsy at this. Sometimes, a kid stops running for a split second, or moves very slowly. The computer thinks, "Oh, this person isn't moving much, so they must be part of the garden!"

As a result, the computer accidentally paints the kid's blurry, ghostly silhouette onto the garden background. Now, your beautiful garden has a weird, transparent ghost of a person stuck in it. This ruins the clarity of the background and makes the moving person look blurry too.

The Solution: MVFusion-GS

The authors created a new system called MVFusion-GS. You can think of this as giving the computer two superpowers to better understand motion:

1. The "Long-Term Memory" (Motion-Variance Guided Refinement)

Imagine you are watching a movie. If you only look at one single frame, you might not know if a car is moving or just parked. But if you look at the whole movie, you see the car drive across the screen.

How it works: The new system doesn't just look at the current moment. It keeps a "scorecard" for every single tiny cloud in the scene. It tracks where that cloud has been over time.
The Analogy: It calculates a "Motion Score." If a cloud wiggles a little bit, moves a lot, or changes size, the system gives it a high score. If it stays perfectly still, it gets a low score.
The Result: Even if a person stops moving for a second, the system remembers, "Hey, this cloud was running around five seconds ago! It's definitely a moving object, not part of the garden." It correctly moves that cloud back to the "Moving" pile, cleaning up the background.

2. The "Context Clue" (MotionFormer Temporal Attention)

Sometimes, motion is tricky. A cloud might look like it's moving one way, but it's actually part of a larger pattern.

How it works: This part acts like a detective looking at the frames immediately before and after the current one. It uses a smart "attention" mechanism (similar to how humans focus on relevant details) to ask, "What are my neighbors doing right now?"
The Analogy: If you see a leaf blowing in the wind, looking at the leaf alone is confusing. But if you look at the branch it's attached to and the other leaves nearby, you understand the whole wind pattern. This module helps the computer understand the "flow" of movement between frames, making the motion look smooth and natural rather than jittery.

The Two Big Wins

By combining these two "superpowers," the paper claims the system achieves two main things:

Cleaner Backgrounds (Distractor-Free): Because the system is so good at spotting even tiny movements, it stops accidentally painting moving objects onto the background. The garden looks perfectly still and clear, with no ghostly ghosts.
Sharper Moving Objects: Because the system correctly identifies what is moving, it can focus its energy on making the moving objects look sharp and detailed, rather than blurring them out.

Summary

In short, previous methods were like a security guard who only checks if someone is moving right now. If they stand still for a second, the guard lets them blend into the crowd.

MVFusion-GS is like a guard who has a memory of the last hour and talks to the guards in the next room. It knows exactly who is a "moving guest" and who is a "static statue," ensuring the background stays clean and the moving guests stay sharp.

The paper tested this on real-world videos with people walking and objects moving, and it consistently produced clearer, higher-quality 3D videos than previous methods.

Technical Summary: MVFusion-GS

1. Problem Statement

Extending 3D Gaussian Splatting (3DGS) to dynamic scenes via deformation fields is central to two tasks: dynamic scene reconstruction (faithfully modeling foreground motion) and distractor-free reconstruction (recovering a clean static background by removing transient objects). While decoupled frameworks like DeGauss separate scenes into static and dynamic branches, they suffer from a critical limitation: a lack of explicit motion awareness.

Standard deformation networks encode latent features through spatio-temporal grids without explicitly modeling the motion patterns of individual Gaussians or their temporal context. Consequently, these networks often fail to capture low-amplitude or transient motions. This leads to "pseudo-static" residuals, where dynamic foreground elements are incorrectly classified as static and left in the background branch. This contamination degrades the quality of the reconstructed static background and results in inaccurate foreground deformation.

2. Methodology: MVFusion-GS

The authors propose MVFusion-GS, a plug-in refinement framework that enhances existing deformation networks with two complementary motion-aware mechanisms. The method treats the baseline deformation network as a coarse predictor and adds a lightweight refinement stage that operates purely in the feature space, requiring no modification to the original deformation heads.

The total deformation is formulated as:
$\Delta \mathcal{G}_d(t) = \Delta \mathcal{G}_d^{\text{base}}(t) + \Delta \mathcal{G}_d^{\text{MR}}(t)$
where the refinement term $\Delta \mathcal{G}_d^{\text{MR}}(t)$ is predicted from a fused feature vector combining the baseline features with two new motion-aware modules.

A. Motion-Variance Guided Refinement (MVG)

This module addresses the lack of global motion cues by aggregating per-Gaussian deformation statistics across time.

Global Trajectory Signature: Periodically, the system samples deformation trajectories for each Gaussian across multiple timesteps. It computes statistics (mean and variance) for position displacement, rotation magnitude, and scale variations (both isotropic and anisotropic). These are aggregated into a 13D global trajectory signature ( $v_i$ ) that serves as a long-term motion intensity prior.
Local Variance Dictionary: To capture short-term variations, a lightweight dictionary caches local motion variance within a sliding temporal window. This provides an instantaneous motion intensity score ( $e_i(t)$ ) for any query time via interpolation.
Injection: These signatures are encoded into the deformation feature space via a lightweight MLP, providing the network with explicit priors to distinguish static Gaussians from true movers.

B. MotionFormer Temporal Attention (MFTA)

While MVG provides global statistics, it lacks short-term temporal interactions. MFTA addresses this by modeling local temporal dependencies.

Temporal Feature Construction: For a Gaussian at time $t_i$ , the module constructs a temporal neighborhood including $t_{i-w}, t_i, t_{i+w}$ . It forms tokens using the deformation feature, predicted deformation, and local motion intensity.
Cross-Attention: A Transformer-based cross-attention mechanism aggregates information from neighboring timesteps, allowing the network to leverage local temporal context for more consistent deformation predictions.
Injection: The resulting temporal motion feature is fused with the baseline and MVG features before being decoded by the deformation heads.

Training Strategy

The training proceeds in three stages:

Base Geometry: Optimizing static geometry with deformation disabled.
Base Deformation: Activating the baseline deformation network to learn fundamental transformations.
Motion-Aware Refinement: Activating the full MVG and MFTA branches, trained end-to-end with zero-initialized weights to refine the deformation updates.

3. Key Contributions

The paper identifies three primary contributions:

A Dual-Task Plug-in Module: A framework that enhances deformation network motion awareness, yielding consistent improvements on both dynamic scene reconstruction and distractor-free reconstruction benchmarks.
Motion-Variance Guided Refinement (MVG): A mechanism that aggregates global trajectory statistics into an explicit motion intensity prior, facilitating more accurate dynamic-static separation.
MotionFormer Temporal Attention (MFTA): A Transformer-based module that leverages cross-attention to model cross-temporal motion dependencies, improving both deformation accuracy and temporal consistency.

4. Experimental Results

The method was evaluated on NeRF On-the-Go (distractor-free), RobustNeRF (distractor-free with cross-view inconsistencies), and Neu3D (dynamic scene reconstruction).

Distractor-Free Reconstruction (NeRF On-the-Go): MVFusion-GS achieved state-of-the-art (SOTA) performance across all averaged metrics (PSNR, SSIM, LPIPS). Notably, it achieved the lowest LPIPS on all six tested scenes, indicating superior perceptual quality. Qualitative results showed the removal of "ghostly" pseudo-static residuals (e.g., blurred pedestrian silhouettes) that persisted in the background of baseline methods like DeGauss.
Dynamic Scene Reconstruction (Neu3D): The method achieved the best mean PSNR and LPIPS. In dynamic regions, it outperformed SOTA methods like SpaceTimeGS and ST-4DGS, preserving sharper structural details in highly dynamic areas (e.g., flames, rapidly moving objects) where competitors exhibited motion blur.
Ablation Studies: Removing either MVG or MFTA resulted in performance drops, confirming that both long-term motion statistics and short-term temporal context are necessary. The full model outperformed variants using only positional variance or self-attention, validating the specific design of the global signature and cross-attention mechanisms.

5. Significance and Claims

The paper claims that explicit motion awareness is the key to improving both foreground motion modeling and static background reconstruction. By correctly re-attributing pseudo-static Gaussians to the dynamic branch, the method yields dual benefits:

Cleaner Backgrounds: Essential for distractor-free reconstruction, eliminating artifacts caused by misclassified dynamic elements.
Finer Foreground Details: Essential for dynamic reconstruction, capturing subtle and transient motions that standard deformation networks miss.

The authors maintain a modest stance in their conclusion, acknowledging that the method still relies on the baseline deformation field to provide reliable initial motion cues. They note that in cases of heavy occlusion or extremely weak motion evidence, where the initial deformation underfits, residual artifacts may remain, pointing to a need for future work on more robust motion-aware reassignment.