💻 computer science

NewtonGS: Physics-Structured Object-Level Neural Newtonian Dynamics for Gaussian Scene Animation

NewtonGS introduces a physics-structured framework that animates static 3D Gaussian scenes by representing objects with a compact 22-dimensional state and a hybrid Gaussian Neural Newtonian Dynamics model, enabling superior state prediction and controllable object-level motion compared to existing methods.

Original authors: Lianlei Shan, Feiyang Ye, Yan Chen, Yong Wu

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lianlei Shan, Feiyang Ye, Yan Chen, Yong Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing with a digital sandbox where you can build entire worlds out of millions of tiny, glowing marbles. These aren't just ordinary marbles; they are "3D Gaussians," a special kind of digital building block that lets computers render scenes so fast and sharp that they look like real life. For a long time, these digital worlds were like dioramas: beautiful to look at, but completely frozen in time. If you wanted to see a ball roll across a table, you had to manually move every single marble, one by one, frame by frame. It was a nightmare for animators and game designers.

Recently, scientists figured out how to make these marbles move, but they mostly treated the whole scene like a giant, wiggly jelly. They could make a character dance or a cloth ripple, but they couldn't easily grab a specific object, say a coffee mug, and tell it, "Hey, roll across the table and bounce off the wall." The computer didn't know the mug had a specific weight, a specific speed, or a specific way it would bounce. It just knew how the pixels looked. To make these digital worlds truly interactive and predictable, we need a way to give every object its own "brain" that understands the laws of physics—like gravity, friction, and momentum—so we can control it like a real toy.

This is where a new method called NewtonGS comes in. Think of it as giving a digital puppet a set of invisible strings that are controlled by the actual laws of physics. The researchers built a system that treats every object in a 3D Gaussian scene not as a messy pile of millions of tiny dots, but as a single, smart character with a specific "state." This state is like a character sheet in a video game, containing exactly 22 pieces of information: where the object is, which way it's facing, how fast it's spinning, how fast it's moving, how big it is, and even how heavy it is and how bouncy it is.

The magic happens because NewtonGS uses a clever mix of old-school physics math and a little bit of machine learning "intuition." It starts with the hard rules of physics (like Newton's laws) to predict how an object should move. But since real life is messy and computers aren't perfect, it adds a "learned residual." Imagine a physics teacher who knows the textbook answers perfectly, but also has a student who has watched a million videos of balls bouncing. The teacher knows the math, but the student knows that sometimes a ball bounces weirdly because of a hidden bump. NewtonGS combines the teacher's math with the student's experience to predict movement that is both physically correct and surprisingly realistic.

The team tested this on a massive set of 32 different types of movements, from simple sliding and rolling to complex bouncing and spinning. They created a digital playground called State-32 with over a million simulated scenarios. In these tests, NewtonGS was able to predict where an object would be, how fast it would be going, and how it would rotate much better than five other methods that relied only on strict math formulas. It made fewer mistakes in predicting the path, the final stopping point, and the speed.

Crucially, the paper shows that this isn't just about numbers. Once the system predicts where the object should be, it instantly updates all the millions of tiny glowing marbles to match that new position. It's like having a conductor who tells every musician in an orchestra exactly when to play their note, ensuring the whole scene moves in perfect harmony. The researchers also showed that this works even when the objects are in different environments, like a toy car on a living room table or a soccer ball in a park.

However, the paper is careful to point out what this system doesn't do yet. It doesn't magically figure out the physics just by looking at a video; it needs to be told the starting speed and weight first. It also can't handle objects breaking apart or changing shape in complex ways (like a piece of clay squishing); it only handles objects that stay solid but might get slightly bigger or smaller. And while it's great at moving one object at a time, it doesn't yet solve the puzzle of two objects crashing into each other and bouncing off one another.

In short, NewtonGS is a significant step forward in making digital worlds feel real. It bridges the gap between static 3D art and interactive physics, allowing us to control digital objects with the same rules we use in the real world. It suggests that by giving digital objects a clear "state" and a physics-based brain, we can create animations that are not only visually stunning but also predictable and controllable, opening the door for better video games, virtual reality, and interactive storytelling.

Technical Summary: NewtonGS

Problem Statement

Animating objects within a static 3D Gaussian Splatting (3DGS) scene requires an explicit object-level dynamic state and a controllable model for object motion. Existing dynamic Gaussian methods primarily focus on reconstructing time-varying scenes or simulating continuous deformations (e.g., 4DGS, deformation fields). These approaches lack a compact, explicit object-level state representation that allows for direct control, prediction, and editing of an existing Gaussian object's motion. Furthermore, purely analytic models often fail to account for model mismatch and unmodeled effects, while unconstrained neural predictors produce motions that are difficult to control and interpret. The core challenge is to represent a Gaussian object with a compact state that retains clear geometric and dynamical meanings, evolve this state over time by capturing both continuous motion and discrete events (like contact), and map the predicted motion back to the Gaussian primitives coherently without assigning separate trajectories to each primitive.

Methodology: NewtonGS

The authors propose NewtonGS, a physics-structured framework for object-level state rollout and Gaussian scene animation. The method operates in three stages: state representation, hybrid dynamics prediction, and state-to-Gaussian transformation.

1. Object-Centric Gaussian State Representation

Each dynamic object $k$ is summarized by a compact 22-dimensional state vector $Z_k$ :
$Z_k = [p_k, q_k, v_k, \omega_k, s_k, u_k, m_k, e_k, \mu_k]$
Where:

$p_k \in \mathbb{R}^3$ : Object center position.
$q_k \in \mathbb{H}$ : Unit orientation quaternion.
$v_k, \omega_k \in \mathbb{R}^3$ : Linear and angular velocities.
$s_k, u_k \in \mathbb{R}^3$ : Aggregate anisotropic scale and its rate.
$m_k, e_k, \mu_k \in \mathbb{R}$ : Mass, restitution, and tangential attenuation (material properties).

This state describes an object aggregate rather than individual primitives. The scale $s_k$ covers size-changing and simple deformation trajectories without introducing a high-dimensional deformation code.

2. Gaussian Neural Newtonian Dynamics (Gaussian-NND)

The core of the framework is a structured hybrid dynamics model that combines an analytic prior with learned residuals.

Continuous Dynamics: Between contacts, the state evolves via a structured vector field integrating translation, quaternion kinematics, gravity, damping, and scale restoration. A neural network $f_\theta$ predicts bounded acceleration residuals $[r_v, r_\omega, r_s]$ to correct the analytic prior. The system is integrated using RK4.
Discrete Contact Events: To handle instantaneous velocity changes at impact, a separate event map detects contact with a horizontal floor ( $y=0$ ). Upon detection, an analytic response (restitution and tangential attenuation) is applied, followed by a learned contact residual network $h_\phi$ that predicts corrections to the post-impact velocity and angular velocity.
Initialization: Both residual networks are initialized to produce zero outputs, ensuring the initial rollout strictly follows the analytic model, with residuals learned from data.

3. State-to-Gaussian Transformation

The predicted object state is converted into a shared affine transformation applied to all Gaussians belonging to the object.

A relative rotation matrix $R_t$ and scale ratio vector $\rho_t$ are computed from the predicted state relative to the initial state.
The Gaussian means $g_i$ and covariances $\Sigma_i$ are updated via a shared transformation matrix $A_t$ :
$g_i^t = A_t(g_i^0 - p_0) + p_t$
$\Sigma_i^t = A_t \Sigma_i^0 A_t^\top$
This ensures the object's internal structure is preserved while the entire cluster moves coherently.

Key Contributions

NewtonGS Framework: An object-level framework that formulates Gaussian animation as the rollout of an explicit 22-dimensional state, followed by a deterministic, shared transformation of the object's Gaussian primitives.
Gaussian-NND Model: A structured hybrid dynamics model combining continuous quaternion/scale evolution, discrete horizontal-floor contact events, and zero-initialized learned residuals. This balances physical interpretability with the ability to correct model mismatches.
Evaluation on Procedural Benchmarks: The introduction of State-32 (for state-rollout evaluation) and Gaussian-32 (for state-to-Gaussian transformation). Experiments demonstrate that NewtonGS achieves lower trajectory, endpoint, and velocity errors than five analytic baselines across both in-distribution and velocity-range-shift splits.

Results

The authors evaluated NewtonGS on the State-32 dataset, which contains 1,048,576 training sequences and 131,072 sequences each for in-distribution (ID) and out-of-distribution (OOD) splits (where OOD involves increased velocity ranges).

State Prediction: On both ID and OOD splits, NewtonGS achieved the lowest Trajectory RMSE, Final Displacement Error (FDE), and Velocity RMSE among five analytic baselines (including Hold-Z0, Constant Velocity, Damped Velocity, Gravity-Bounce, and Physics-Prior).
- On ID, NewtonGS reduced trajectory, endpoint, and velocity errors by 3.61%, 2.57%, and 2.34% relative to the best baseline.
- On OOD, reductions were 3.06%, 5.25%, and 2.27%, respectively, indicating robustness to velocity shifts.
Cross-Representation: On the Gaussian-32 and Same-32 subsets, NewtonGS outperformed the L4GM video-to-4DGS method in trajectory, endpoint, and quaternion errors, demonstrating effective conversion from predicted states to animated Gaussian objects.
Video Proxy Comparison: In a "Common-12" comparison against video generation models (Sora, Veo 3, Wan, etc.), NewtonGS achieved higher scores in physical consistency (PISp), background stability (BCp), and motion smoothness (MSp), with lower variance across motion categories.

Significance and Claims

The paper claims that NewtonGS provides a necessary bridge between static 3D Gaussian representations and controllable, physics-guided motion. By assigning explicit dynamic states to Gaussian objects, the method enables predictable and editable motion, which is crucial for scene animation and interactive content creation.

The authors emphasize that their approach differs from existing 4DGS methods by favoring a coherent object frame and direct state-space control over independently predicted primitive motion. The structured hybrid dynamics allow the model to capture diverse motion patterns (ballistic, bouncing, rolling, oscillatory) that fixed analytic rules fail to capture, while maintaining interpretability through the underlying physical priors.

The paper remains modest regarding its scope:

It focuses on object-level dynamics and does not model articulated joints, topology changes, or complex multi-object interactions (e.g., pairwise collisions are not explicitly modeled in the event map).
The current implementation handles contact with a horizontal floor; wall and slope contacts are learned via residuals rather than explicit solvers.
The reported results start from labeled initial states; the paper does not claim to solve the automatic observation-to-state inference problem from raw video, though it provides deterministic interfaces for grouping and state lifting.
The evaluation is conducted on procedurally generated synthetic datasets; while the method demonstrates effective animation, it does not claim to achieve photorealistic 4D reconstruction or superiority to text-to-video systems in general generation tasks.