Technical Summary: Learning 3D-Gaussian Simulators from RGB Videos

Problem Statement

Realistic simulation is a cornerstone for embodied intelligence, yet traditional analytical simulators require precise geometry, poses, and material parameters, making them impractical for arbitrary scenes. While 2D video generation models have demonstrated striking visual realism, they often fail to maintain spatial consistency, object permanence, and physical plausibility due to a lack of 3D structure awareness. Conversely, existing 3D particle-based simulators often rely on privileged signals (e.g., depth sensors, object tracks) or hand-crafted graph constructions (e.g., kNN graphs), which hinder scalability and generalization to data-rich regimes.

The paper addresses the challenge of learning a generalizable, scalable visuo-physical simulator directly from raw multi-view RGB videos without relying on privileged signals or manually designed graph biases.

Methodology: 3DGSim

The authors propose 3DGSim, a fully end-to-end differentiable framework that learns 3D particle-based simulators from multi-view RGB videos. The pipeline consists of three jointly trained modules:

1. View-Independent Inverse Renderer

To overcome the limitations of view-dependent encodings, 3DGSim extends MVSplat (a feed-forward 3D reconstruction method).

• Feature Extraction: It extracts pixel-aligned features from multi-view RGB history.
• View Independence: A learned feature transformation module, utilizing FiLM conditioning on depth, pixel shift, density, and ray geometry (Plücker coordinates), maps these features into a unified, view-independent latent particle representation.
• State Representation: The state of each particle is distilled into a compact form $\tilde{g}_i = (p_i, f_i)$, where $p_i$ is the position and $f_i$ is a visuo-physical latent feature. The feature $f_i$ is decomposed into an invariant part ($f^{inv}$, encoding shape/appearance) and a dynamic part ($f^{dyn}$, encoding motion properties). Only $f^{dyn}$ is updated during simulation.

2. Transformer-Only Dynamics Engine

Instead of using Graph Neural Networks (GNNs) with kNN-based connectivity and hand-crafted edge features, 3DGSim employs a Transformer-only architecture based on PTv3.

• Serialization: To handle unstructured, time-varying point clouds, the method serializes particles using a space-filling curve (SFC) to create a spatial code.
• Temporal Encoding: A temporal code is appended to the spatial code to form a 64-bit serialization key.
• Temporal Merging (TEM): A novel module applies a one-bit right shift to temporal codes, allowing points from adjacent timesteps to be grouped into the same attention patches. This enables the model to aggregate spatiotemporal information without training separate models for each timestep or using abstract temporal nodes.
• Prediction: The transformer predicts position deltas ($\Delta p$) and dynamic feature deltas ($\Delta f^{dyn}$) for the next timestep.

3. Gaussian Splatting Decoder

The predicted particle states are decoded into 3D Gaussian Splatting parameters (position, rotation, scale, opacity, spherical harmonics) via a learned head.

• Rendering: The system renders novel views using differentiable tile-based rasterization.
• Loss Function: The model is trained end-to-end using an image reconstruction loss ($L_2$ and LPIPS) computed on both past (reconstruction) and future (prediction) frames. A temporal decay factor is applied to future frames to prioritize short-term accuracy while encouraging long-horizon stability.

Key Contributions

1. Inverse Renderer: Extends MVSplat to fuse pixel-aligned features into a particle-based visuo-physical latent representation that is invariant to camera viewpoint.
2. Temporal Encoding & Merging: Introduces a hierarchical module that processes arbitrary timesteps by merging temporal codes, eliminating the need for abstract temporal nodes or per-step models.
3. Transformer-Only Dynamics: Removes graph biases (kNN, hand-crafted edges) in favor of space-filling curve ordering and learned spatiotemporal embeddings, significantly reducing computational overhead.
4. End-to-End Differentiability: Connects inverse rendering, transformer dynamics, and Gaussian splatting into a single framework optimized via image reconstruction loss.
5. Open Source: The authors release code and datasets to establish a reproducible baseline.

Experimental Results

The model was evaluated on three datasets: Rigid (MOVI/GSO objects), Elastic (plastic deformable objects), and Cloth (fixed-corner deformations).

• Performance: 3DGSim achieves competitive long-horizon simulation accuracy (up to 80 steps) compared to state-of-the-art baselines like Cosmos (NVIDIA et al., 2025) and 3D baselines like Spring-Gauss.
• Ablation Studies:
  • Temporal Merging: Removing the TEM module caused a catastrophic performance drop (from ~33 PSNR to ~18 PSNR), proving its necessity for temporal reasoning.
  • Latent vs. Explicit: A latent implicit representation outperformed explicit 3DGS primitives, suggesting that abstracting physical properties (mass, friction) into the latent space improves generalization.
  • Context Window: A 4-step input window performed better than a 2-step window, indicating the need for longer temporal context to infer velocity and acceleration.
• Generalization & Scene Editing:
  • Out-of-Distribution (OOD): Despite being trained only on single-object collisions, 3DGSim successfully predicted multi-object interactions and scenarios with ground removal (freefall), whereas 2D baselines (CosmosFT) hallucinated levitating objects or merged distinct entities.
  • Lighting: The model captured emergent properties like shadows by altering ground particle appearance.
• Real-World Application: Preliminary results on the HO-Cap dataset (hand-object manipulation) demonstrated the ability to simulate 120-step interactions using only initial frames and action conditioning, despite the lack of a large passive multi-view dataset.

Significance and Claims

The paper claims that 3DGSim represents a significant step toward scalable, data-driven visuo-physical simulation. By leveraging the efficiency of Gaussian Splatting and the relational inductive biases of Transformers (via space-filling curves) rather than GNNs, the method bridges the gap between 2D video generation and 3D physical simulation.

The authors emphasize that 3DGSim offers spatial causality, ensuring interactions are restricted to spatially grounded particles. This provides robustness to OOD inputs and maintains object permanence, contrasting with 2D models where dynamics can emerge from generative flexibility rather than physical laws. While the 3D formulation imposes stricter structural constraints (making it harder to dynamically create/destroy particles compared to 2D generators), the authors argue this trade-off yields greater stability and interpretability for embodied intelligence tasks.

The work positions itself as a foundation for future research into vision-language-driven simulators and large-scale validation on real-world multi-view datasets, potentially mitigating the need for multi-view inputs through advances in monocular inverse rendering.