💻 computer science

DyG $^2$ T: Modeling Object Dynamics with 3D Gaussian Temporal-Spatial Particle Graph Transformer

The paper proposes DyG $^2$ T, a novel framework that enhances object motion trajectory prediction by spatially enriching Key Points with raw particle details, temporally disentangling frame variations to capture discriminative evolution, and leveraging a Particle Graph Transformer for robust multi-scale interaction modeling.

Original authors: Yansong Wang, Zhaobo Qi, Xinyan Liu, Beichen Zhang, Shuhui Wang, Weigang Zhang, Qingming Huang

Published 2026-08-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yansong Wang, Zhaobo Qi, Xinyan Liu, Beichen Zhang, Shuhui Wang, Weigang Zhang, Qingming Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

To understand how a robot might one day catch a falling apple or a digital twin might simulate a collapsing building, we must first solve a problem that has long stumped computers: predicting how objects move when they are not rigid. In the physical world, most things are not solid blocks of steel; they are soft, stretchy, and full of internal complexity. When a piece of fruit drops, or a toy bounces, its surface ripples and its shape changes in ways that depend on its hidden material properties. For a machine to interact with such objects, it cannot simply memorize a single path. It must understand the invisible forces traveling through the object's interior, inferring how a push on one side will ripple to the other. This requires a system that can watch a few seconds of video, reconstruct the object's three-dimensional shape in real-time, and then guess where every part of that shape will be a moment later.

For years, researchers have tried to teach computers this skill by breaking objects down into a sparse set of points, like a skeleton made of a few dozen dots. The computer would then try to guess how these dots move based on their neighbors. However, this approach often fails because it throws away too much information. By focusing only on the few dots, the system loses the fine-grained details of the object's surface and the subtle geometric relationships between different parts. The result is a prediction that drifts off course, where the object's shape blurs or its path wanders away from reality. The researchers behind a new study, published in the IEEE Transactions on Circuits and Systems for Video Technology, have developed a method to fix this. They call their system DyG2T, and it works by refusing to ignore the details that previous methods discarded.

Instead of looking at just a few key points, the new system starts by reconstructing the entire object as a cloud of thousands of tiny, trackable particles. It then selects a smaller group of "key points" to act as anchors, but unlike older methods, it does not let these anchors float in isolation. The system actively reaches out to the surrounding cloud of particles to gather local details, effectively filling in the gaps between the anchors. It also pays close attention to the relative positions of the anchors themselves, ensuring the computer understands the object's structure, not just the location of its parts. This process is akin to a sculptor who, instead of just marking a few points on a block of clay, constantly checks the texture and shape of the clay between those points to ensure the final form is accurate.

The researchers found that simply having more data was not enough; the computer also needed to understand how the object changes over time without getting confused. In previous attempts, the computer often mixed up the features of the object at one moment with the features of the next, causing the prediction to collapse into a blur. To solve this, the team introduced a process that separates the changes happening from one frame to the next. They trained the system to identify the specific differences that matter most, amplifying the signal of movement while filtering out the noise. This allows the computer to see the object's evolution clearly, distinguishing between a slight wobble and a major shift in direction.

Once the system has a clear, detailed, and time-separated understanding of the object, it uses a powerful network to predict the future. This network looks at the entire object at once, rather than just checking neighbors one by one. By considering the relationship between every part of the object simultaneously, it can model complex interactions that happen across long distances within the material. For example, if one side of a bouncing ball is compressed, the system understands how that pressure travels instantly to the other side, rather than waiting for a chain reaction of local steps. This global view prevents the prediction from becoming "homogenized," or washed out, which was a common failure in earlier models.

The team tested their method on both computer-generated simulations and real-world videos of toys and elastic objects being dropped and bouncing. In the simulations, where the ground truth is known exactly, their system predicted the motion of objects like bananas, apples, and toruses with significantly higher accuracy than existing methods. It reduced the error in the predicted path by a large margin and produced visuals that looked much sharper and more realistic. When they moved to real-world footage, the system continued to perform well, handling complex shapes and materials that had never been seen before. It even managed to predict the behavior of objects made of mixed materials, such as a rigid frame holding soft elastic parts, a scenario that often confuses other systems.

The researchers also checked how well their system held up when the input data was imperfect or noisy, which is common in real-world cameras. Even with slight distortions or missing information, the system maintained its accuracy, suggesting that its method of gathering and organizing information is robust. They further verified that the system respected the laws of physics, ensuring that the predicted movements were consistent with how real objects stretch, compress, and accelerate. The study concludes that by combining a detailed spatial reconstruction with a careful separation of time-based changes, it is possible to create a much more reliable model of object dynamics. This work suggests a path forward for machines that need to interact with the messy, changing physical world, moving beyond simple guesses to a deeper, more accurate understanding of how things move.

Technical Summary: DyG2T - Modeling Object Dynamics with 3D Gaussian Temporal-Spatial Particle Graph Transformer

1. Problem Statement

The paper addresses the fundamental challenge of modeling object dynamics and predicting motion trajectories from limited visual observations, particularly for embodied interaction scenarios. Existing approaches typically reconstruct 3D particle representations from 2D videos and then downsample them into sparse "Key Points" (via Farthest Point Sampling) for dynamic modeling. However, the authors identify two critical limitations in current methods:

Loss of Fine-Grained Details: Compressing raw particles into sparse Key Points discards local geometric and appearance details. Relying solely on coordinates weakens the model's perception of object structure.
Feature Homogenization and Temporal Entanglement: Locality-constrained message passing (common in Graph Neural Networks) leads to feature homogenization, blurring distinctions between particles. Furthermore, without explicit temporal disentanglement, features from different frames tend to collapse into overlapping distributions, reducing temporal discriminability and causing trajectory drift and inaccurate appearance prediction.

These issues result in drifting trajectories and poor appearance reconstruction, especially for elastic objects with complex, material-dependent behaviors that are difficult to observe from sparse viewpoints.

2. Methodology: DyG2T

The proposed framework, DyG2T, infers object motion by spatially completing and temporally disentangling Key Point representations while modeling multi-scale interactions via a Particle Graph Transformer. The pipeline consists of three main stages:

A. Dynamic Reconstruction

The system first utilizes 3D Gaussian Splatting (3DGS) to reconstruct trackable raw particle sequences from multi-view 2D observations.

Initialization: Vanilla 3DGS is applied to the initial frame to obtain 3D Gaussian parameters.
Tracking: For subsequent frames, appearance descriptors are frozen, while spatial descriptors are optimized under multi-view constraints, yielding temporally trackable 3D position sequences (Raw Point Clouds).

B. Spatial-Temporal Feature Completion and Aggregation

This module enhances particle representations from two complementary perspectives before dynamics modeling:

Particle-level Spatial Semantic Completion:
- Coordinate Encoding: Key Point coordinates are mapped to features via a Coord Net (MLP).
- Local Detail Aggregation: A PointNet encodes the positions of $k$ -nearest raw particles for each Key Point, using max-pooling to aggregate fine-grained local appearance information.
- Geometric Structure Encoding: A PosDiff Encoder models pairwise relative offsets between Key Points to capture directed layout relations.
- Fusion: A Multi-head Position-Aware Attention mechanism fuses coordinate features, neighborhood features, and relative offset embeddings. A Gaussian-weighted relation prior regularizes the attention distribution to adhere to geometric correlations.
Object-level Dynamic Temporal Aggregation:
- Temporal Disentangling Network (TDN): To prevent semantic collapse across frames, the TDN estimates dominant cross-frame disentangling offsets in the latent space. Using a 3-frame window (where $t-1$ is the reference), it amplifies inter-frame differences to produce temporally discriminative features.
- Temporal Attention: These disentangled features are aggregated via Temporal Attention, which assigns frame-wise importance scores to capture evolution cues while suppressing redundancy.

C. Dynamics Modeling via Particle Graph Transformer

To model interactions from local contacts to global structure, DyG2T employs a Particle Graph Transformer:

Graph Construction: A graph is constructed where Key Points are nodes, connected to their top- $k_G$ nearest neighbors within a distance threshold.
Global Attention: Unlike local message passing, the Transformer uses global attention to capture long-range dependencies directly. This mitigates representation homogenization.
Gated Residuals: Gated residual connections are introduced between layers to retain informative signals and prevent over-smoothing.
Prediction: The output features are decoded via an MLP to predict the displacement field ( $M^*,t$ ) for Key Points. These displacements are integrated to update Key Point positions.
Densification: Finally, Linear Blend Skinning (LBS) interpolates the predicted Key Point motions to the raw particles to generate the full 3D Gaussian parameters for future frames.

3. Key Contributions

The paper claims the following contributions:

DyG2T Framework: A dynamics modeling framework that mitigates trajectory prediction bias by capturing differentiated multi-scale dependencies within particle representations.
Spatiotemporal Enrichment: The introduction of a spatial completion mechanism (aggregating raw particle details and relative offsets) and a temporal aggregation mechanism (TDN and Temporal Attention) to create robust, discriminative embeddings.
Multi-Scale Interaction Modeling: The use of a Particle Graph Transformer to preserve discriminative features and model long-range dependencies, overcoming the limitations of locality-constrained methods.
Generalization: Demonstrated strong cross-object generalization and real-world scalability on both synthetic and real-world datasets.

4. Experimental Results

The authors evaluated DyG2T on the Spring-Gaus dataset (synthetic and real-world subsets) and a custom Unity3D-Heterogeneous dataset.

Dynamic Reconstruction: DyG2T achieved lower Chamfer Distance (CD) and Earth Mover's Distance (EMD) compared to the baseline Spring-Gaus, indicating high-fidelity 3D trajectory recovery.
Dynamics Modeling & Reasoning:
- Synthetic Data: DyG2T outperformed baselines (Spring-Gaus, GS-Dynamics) in 3D trajectory metrics (CD, EMD) and perceptual image quality (LPIPS). It showed superior generalization across diverse geometries (e.g., Banana, Cross, Apple).
- Real-World & Cross-Benchmark: On real-world toys and the Unity3D-H dataset, DyG2T maintained high PSNR, SSIM, and low LPIPS, outperforming GS-Dynamics on complex geometries like the "Burger."
- Heterogeneous Objects: On a Heterogeneous Torus, DyG2T surpassed strong physics-prior baselines (PAC-NeRF, GIC) in trajectory accuracy and physical consistency metrics (Local Stretch Error, Strain Pattern Correlation, Acceleration Consistency Error).
Robustness: The model demonstrated resilience to noise and non-rigid distortions, maintaining performance where baselines degraded significantly.
Ablation Studies:
- Removing spatial completion or temporal disentangling caused substantial drops in accuracy, validating the necessity of both modules.
- Replacing the Transformer with a vanilla GNN led to significant errors, confirming the need for global attention to capture long-range dependencies.
- A 3-frame observation window was found optimal; extending it to 5 frames introduced instability.

5. Significance and Limitations

Significance:
The paper positions DyG2T as a solution that bridges the gap between physics-regularized reconstruction and differentiable dynamics prediction. By explicitly addressing the loss of local details and the homogenization of features, it enables accurate future trajectory prediction and appearance decoding for elastic objects, which is crucial for embodied intelligence and digital twins. The framework achieves a balance between physical plausibility and generalization to unknown materials without relying on predefined constitutive models.

Limitations:
The authors modestly acknowledge specific constraints:

Fine-Grained Deformation: The LBS-based densification may attenuate deformations at spatial scales smaller than the Key Point spacing (e.g., localized wrinkles).
Appearance Modeling: The method reuses static appearance descriptors (Spherical Harmonics) and does not explicitly model time-varying or view-dependent changes, which could reduce fidelity under strong specular reflections or dynamic lighting.

Future work is suggested to include residual deformation modeling and time-varying appearance modeling to address these fidelity issues.