💻 computer science

LagrangeGS: Non-Conservative Lagrangian System on Dynamic 3D Gaussian Splatting

LagrangeGS introduces a non-conservative Lagrangian framework for Dynamic 3D Gaussian Splatting that resolves issues of physical inconsistency, irreversibility, and geometric collapse through identity-matrix approximation, time-independent force constraints, and local rigid alignment, thereby enabling stable long-term extrapolation and counterfactual physics-based editing.

Original authors: Shogo Sato, Takuhiro Kaneko, Shoichiro Takeda, Tomoyasu Shimada, Riku Inoue, Kazuhiko Murasaki, Ryuichi Tanida

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shogo Sato, Takuhiro Kaneko, Shoichiro Takeda, Tomoyasu Shimada, Riku Inoue, Kazuhiko Murasaki, Ryuichi Tanida

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to capture a moving scene, like a fan spinning or a ball falling, not just as a flat video, but as a three-dimensional world that you can walk around inside. For years, scientists have developed ways to build these digital worlds from photos, creating static scenes that look incredibly real. More recently, they learned to make these worlds move, allowing a camera to fly through a video of a bustling room or a rotating object. However, these moving digital worlds have a major weakness: they are essentially clever tricks that memorize what happened in the video. If you ask the computer to show you what happens a second after the video ends, or to rewind the video to show what happened before it started, the digital world often falls apart. The objects might stretch into strange shapes, vanish, or move in ways that defy the laws of physics, because the computer never actually learned how gravity or momentum works; it only learned how the pixels changed.

A team of researchers at NTT in Japan has introduced a new approach called LagrangeGS that fixes this by teaching the digital world to obey the rules of physics. Instead of just watching how the scene changes, their system describes the movement using a fundamental framework known as Lagrangian mechanics. In simple terms, this framework treats every moving part of the scene as a physical object with mass, energy, and forces acting upon it. By doing this, the computer doesn't just guess what comes next; it calculates the future based on how energy is stored and how forces push and pull. This allows the system to predict what a scene will look like far into the future, rewind time to see the past, or even change the rules of the scene, such as making gravity weaker, without needing to be retrained on new data.

The researchers built their system on top of a technology that represents a scene as millions of tiny, colored, fuzzy spheres called Gaussian particles. In previous methods, these particles were allowed to drift and deform freely to match the video, which worked well for the time the video was recorded but failed miserably when the system tried to guess what happened next. The new method forces these particles to behave like a physical system. To make this possible for millions of particles, the team simplified the complex math usually required to calculate how these particles interact. They treated every particle as if it had the same simple weight and moved independently, which removed a massive computational barrier. They also ensured that the forces acting on the particles did not change arbitrarily over time, a key requirement that allows the system to run backward in time just as easily as it runs forward.

To keep the digital objects from falling apart, the researchers added a clever constraint that keeps groups of particles moving together as if they were rigid parts of a solid object, like the blades of a fan or the body of a ball. This prevents the particles from scattering into a cloud of noise when the system tries to simulate time far beyond the original video. When they tested this new system on scenes featuring a spinning fan and a falling ball, the results were striking. While older methods quickly turned the spinning fan into a blurry mess or made the falling ball explode into a giant cloud of pixels, the new system kept the shapes intact and the motion smooth, even when predicting time steps that were three times longer than the original video.

One of the most compelling demonstrations of this work is the ability to reverse time. Because the system is built on physical laws that do not care which way time flows, the researchers could simply tell the computer to run the simulation backward. The result was a perfect rewind of the scene, where the ball rolled back up to its starting height and the fan spun in reverse, with the particles returning exactly to their original positions. This is something previous systems could not do; they would simply fail or produce a jumbled mess when asked to go backward. The researchers also showed that they could edit the physics of the scene on the fly. By adjusting a single setting, they could make the ball fall much slower, as if it were on the moon, or much faster, as if gravity were stronger. They did this not by retraining the model, but by simply changing the numbers that defined the forces in the simulation.

The study confirms that by grounding these digital representations in the actual laws of physics, it is possible to create dynamic 3D worlds that are stable, reversible, and editable. The researchers found that while their method sometimes produced slightly less sharp images than the most advanced purely visual methods in very specific indoor scenes, it completely eliminated the geometric collapse that plagued other systems during long simulations. They also noted that their current approach works best when the objects in the scene do not crash into each other or collide in complex ways, as the system treats the particles as moving independently. Despite this limitation, the work marks a significant step forward, transforming dynamic 3D scenes from static visual recordings into living, physical models that can be explored, reversed, and manipulated with the same logic we use to understand the real world.

Technical Summary: LagrangeGS

Problem Statement
Dynamic 3D Gaussian Splatting (3DGS) has achieved photorealistic reconstruction of time-varying scenes, yet existing physics-aware extensions suffer from fundamental limitations. While recent methods predict velocity or acceleration fields to enable extrapolation, they merely fit vector fields to visual deformations without satisfying Lagrangian mechanics. This lack of physical grounding leads to three critical issues:

Physically inconsistent trajectories: Motion does not adhere to conservation laws or energy principles.
Lack of time-reversibility: The systems cannot consistently reconstruct past states from current observations.
Geometric collapse: Long-term extrapolation often results in unbounded volume divergence or indistinct haze.

Furthermore, applying standard Generalized Lagrangian Neural Networks (G-LNNs) directly to dynamic 3DGS is computationally intractable. It requires the inversion of a massive velocity-Hessian matrix ( $\partial^2 L / \partial \dot{q}^2$ ) for millions of Gaussian particles at every integration step. Additionally, standard G-LNNs often employ time-dependent forces, which mathematically preclude exact time reversal.

Methodology
The authors propose LagrangeGS, a framework that formulates dynamic 3DGS as a non-conservative Lagrangian system parameterized by potential energy ( $U$ ), initial velocity ( $v_0$ ), and non-conservative forces ( $Q$ ). The architecture consists of three core components:

Time-independent Decoupled LNN (TD-LNN):
To resolve the computational bottleneck of Hessian inversion, the method approximates the velocity-Hessian as an identity matrix. This is physically equivalent to assigning a uniform unit mass to all Gaussian particles and decoupling their inertial dynamics. The kinetic energy is simplified to $T(\dot{q}) = \frac{1}{2}\|\dot{q}\|^2$ .
To enable time reversal, the non-conservative force is restricted to be explicitly time-independent, $F(q, \dot{q}, t) = Q(q, \dot{q})$ . This allows the system to be integrated backward in time using a negative time step, ensuring mathematical well-posedness for state recovery. The resulting acceleration for each particle $j$ is a closed-form solution:
$\ddot{q} = -\nabla_q U(q) + Q(q, \dot{q})$
Both $U$ and $Q$ are parameterized by Multi-Layer Perceptrons (MLPs).
Local Rigid Alignment (LRA):
To address geometric collapse caused by decoupled particle dynamics, the method introduces a regularization step. For each particle, a local rigid support is defined using its $k$ -nearest neighbors in the initial state. During rollout, the candidate positions predicted by the TD-LNN are aligned to a locally rigid motion of this support using a differentiable Kabsch solver. This binds particles into locally rigid structures while allowing for global continuous deformations.
Three-Stage Optimization:
- Stage 1 (Trajectory Distillation): The TD-LNN is trained to minimize the Mean Squared Error (MSE) between its integrated trajectories and trajectories extracted from a pre-trained dynamic 3DGS model (e.g., DefGS or FreeGave). A penalty term encourages the dynamics to remain conservative where possible.
- Stage 2 (TD-LNN Fine-tuning): The model is fine-tuned using photometric loss (SSIM and $L_1$ ) on the rendered images. Gradients flow through the differentiable Kabsch alignment to the TD-LNN parameters, allowing the model to escape overfitting to the source trajectories.
- Stage 3 (Gaussian Parameter Finetuning): The physics parameters are frozen, and the initial Gaussian parameters (position, rotation, scale, opacity, color) are refined to absorb residual visual errors.

Key Contributions

Lagrangian Formulation: The first framework to explicitly formulate dynamic 3DGS as a non-conservative Lagrangian system, fundamentally solving physically inconsistent trajectories and defining a theoretical basis for physics manipulations.
TD-LNN Architecture: A novel neural architecture that approximates the velocity-Hessian as an identity matrix for computational tractability and employs time-independent forces to enable consistent time reversal.
Local Rigid Alignment (LRA): A mechanism that prevents geometric collapse during long-term extrapolation by binding decoupled particles into locally rigid structures.
Physics Manipulation: The ability to perform counterfactual physics-based editing (e.g., scaling gravity or initial velocity) and time reversal at inference time without retraining.

Results
Experiments were conducted on the Dynamic Object and Dynamic Indoor Scene benchmarks, comparing LagrangeGS against deformation-based methods (T-NeRF, DefGS) and physics-aware baselines (NVFi, TRACE, FreeGave).

Extrapolation: LagrangeGS demonstrates stable long-term extrapolation. While DefGS suffers from unnatural motion and FreeGave exhibits geometric drift and volume divergence, LagrangeGS maintains physically consistent and smooth motion. Quantitatively, LagrangeGS improves PSNR by +5.05 on the Dynamic Object dataset when distilling from DefGS. When distilling from FreeGave, it shows a marginal drop in photometric metrics but significantly improves geometric stability (preventing volume expansion from 1.9 $\times$ to 1.1 $\times$ ).
Time Reversal: The system successfully recovers past states ( $t < 0$ ) with high consistency. A forward-backward cycle test showed a mean position error of only 1.3% (Dynamic Object) and 0.19% (Dynamic Indoor Scene).
Counterfactual Editing: The framework allows for explicit manipulation of physical parameters. Scaling the learned potential field ( $U$ ) results in predictable changes to falling time that closely match the theoretical relation $t \propto \beta^{-0.5}$ ( $R=0.999$ ).
Ablation: Removing LRA leads to geometric collapse in long-term rollouts. Removing the non-conservative force $Q$ degrades performance on scenes with non-conservative motion (Dynamic Object) but slightly improves scores on strictly conservative scenes (Indoor), confirming the model's adaptability.

Significance
The paper claims that LagrangeGS establishes an explicit formulation of dynamic scenes as Lagrangian systems, bridging the gap between visual fidelity and physical plausibility in 3DGS. By solving the computational intractability of standard LNNs and enforcing time-independence, the work enables stable long-term extrapolation, consistent time reversal, and interpretable physics-based editing. This paves the way for physically grounded dynamic modeling and manipulation, which are critical for downstream applications such as robot motion planning and embodied AI. The authors note limitations, including the reliance on pre-trained kinematic trajectories for initialization and the assumption of decoupled dynamics, which may not capture complex collisions, suggesting these as directions for future work.

Technical Summary: LagrangeGS

More like this