Technical Summary: Scene-Level Heterogeneous Physics Simulation with 3D Gaussian Splats

1. Problem Statement

Despite the state-of-the-art (SOTA) photorealistic rendering capabilities of 3D Gaussian Splatting (3DGS), these assets remain largely "dead" in terms of physical interactivity. A fundamental representation gap exists:

• 3DGS Representation: Defined by covariance ($\Sigma$), spherical harmonics ($C$), and opacity ($\sigma$), optimized for rendering but incompatible with production-grade physics engines.
• Physics Engine Limitations: Industrial engines (e.g., Unreal Engine, Houdini) operate on particles, meshes, and rigid bodies using heterogeneous solvers (MPM, SPH, PBD). They cannot natively understand 3DGS.

Prior attempts to bridge this gap have followed a "siloed" paradigm, creating monolithic, proprietary physics solvers specifically for 3DGS. These approaches suffer from two critical flaws:

1. Homogeneity: They create closed systems where only GS particles exist, preventing interaction with standard Computer Graphics (CG) assets like meshes, fluids, or rigid bodies.
2. Isolation: They operate in voids or on ideal planes, failing to support complex, pre-existing 3D static collision geometry derived from real-world scene captures.

Consequently, prior methods are limited to "toy" scenarios and cannot achieve scalable, scene-level, heterogeneous physics simulations.

2. Methodology: Representation Abstraction Framework (RAF)

The authors propose a novel Representation Abstraction Framework (RAF) that acts as a bridge, unifying diverse assets into a single physical simulation context without reinventing physics solvers. The pipeline consists of three stages:

Stage 1: Asset Preparation and Abstraction

The goal is to translate heterogeneous inputs into two unified components: a static collision mesh ($M_{static}$) and a unified particle list.

• Static World ($M_{static}$): A composite mesh assembled from captured real-world data (processed via Gaussian segmentation, object-scene restoration, and surface reconstruction) and optional imported assets. This defines the non-moving collision boundaries.
• Dynamic World (Unified Particle List): All dynamic assets are abstracted into a single particle set ($X$):
  • Captured 3DGS: Converted into physical particles ($X_{gs}$) by sampling a continuous opacity field. Each particle $p_j$ stores kinematic state ($x, v, m$), model type, and constitutive parameters (e.g., Young's modulus). Visual data ($\Sigma_{rest}, C, \sigma$) is cached separately.
  • Virtual Meshes: "Particle-ized" via Poisson disk sampling. Barycentric weights are pre-computed to bind the template mesh to the particles.
  • External Particles/Fluids: Loaded directly as initial positions with assigned solver types (e.g., SPH).
  • Kinematic Assets: Treated as rigid bodies with user-defined transforms; they act as one-way colliders.

Stage 2: Unified Simulation Kernel

A modular, solver-agnostic kernel processes the unified particle list and $M_{static}$.

• Heterogeneous Solvers: The kernel interfaces with specialized solvers (MPM for soft bodies, SPH for fluids, PBD for cloth, rigid-body, and articulated-body dynamics).
• Coupling Mechanism: The core capability is robustly managing interactions between disparate solvers and static geometry.
  • Continuous Coupling: Handles force exchanges (e.g., fluid pressure on soft bodies).
  • Discrete Coupling: Uses a unified impulse-based mechanism to resolve collisions and friction between any entities (particle-mesh, rigid-rigid) using coefficients of restitution ($e$) and Coulomb friction ($\mu$).
• Execution: Solvers execute iteratively in a specific sequence (Rigid/Articulated $\to$ MPM $\to$ SPH $\to$ PBD), propagating updated states between solvers to resolve constraints within a single time step.

Stage 3: Heterogeneous Recoupling and Rendering

The sparse physical results are "reverse-translated" back into high-fidelity visual assets.

• 3DGS Assets: The new particle position $x'_j$ updates the Gaussian center. The physical deformation gradient $F'$ is applied to the cached rest-state covariance ($\Sigma_{rest}$) to compute the new covariance ($\Sigma' = F' \Sigma_{rest} F'^T$). Rotation from $F'$ updates Spherical Harmonic coefficients.
• Mesh Assets: New vertex positions are computed by applying pre-computed barycentric weights to the updated particle positions.
• Rendering: Serialized assets (.splat, .abc, .pbd) are loaded into Unreal Engine 5. The framework leverages UE5's native capabilities, including Lumen global illumination and ray-traced shadows, to render the co-simulated scene.

3. Key Contributions

1. First Scene-Level Heterogeneous Framework: The first system to enable 3DGS assets to participate in scene-level, multi-solver physics simulations alongside standard CG assets.
2. Representation Abstraction Layer: A novel layer that translates diverse dynamic assets (3DGS, meshes, fluids) and complex static scenes into a single physical particle context, enabling two-way interactions.
3. End-to-End Pipeline: A complete pipeline demonstrating complex interactions (e.g., fluid splashing on deformable 3DGS, cloth draping on captured statues) rendered with photorealistic fidelity in Unreal Engine 5.
4. Overcoming "Siloed" Limitations: The framework moves beyond isolated, ideal-plane simulations to support complex, pre-existing 3D static geometry and heterogeneous asset coupling.

4. Results and Evaluation

The framework was validated on five complex scenarios unattainable by prior "siloed" methods:

• Fluid on Virtual Mesh: SPH fluid interacting with an imported mesh bowl and a static 3DGS garden scene.
• Fluid on GS Soft Body: High-viscosity SPH fluid coupling with a captured 3DGS soft-body donut (MPM), demonstrating two-way deformation.
• Articulated Dynamics: A kinematically driven robotic arm manipulating a rigid body on a table.
• Complex Geometry Interaction: PBD cloth realistically draping and sliding over a captured 3DGS statue with non-convex geometry.
• Rigid-Body in 3DGS Container: Rigid bodies interacting within a captured 3DGS basket.

Ablation Studies:

• Unified Kernel: Removing the coupling mechanism caused fluids and soft bodies to pass through each other, confirming the necessity of the heterogeneous solver interface.
• Scene-Level Geometry: Removing the static collision mesh ($M_{static}$) forced cloth simulations to default to an "ideal plane," causing them to pass through captured objects.
• Representation Abstraction: Removing the recoupling step resulted in only sparse simulation particles being visible, lacking photorealistic appearance.

Comparison with Photogrammetry:
When compared to standard photogrammetry (KiriEngine) for mesh extraction, the authors' 3DGS-based pipeline preserved sharp geometric details (e.g., bulldozer treads, table slats) that were lost or "blobby" in the mesh baseline, while still enabling physics simulation.

5. Significance and Claims

The paper claims to fundamentally transform 3DGS from "passive renderables" into "active participants" in physical sandboxes. By bridging the representation gap, the work enables:

• Dual Fidelity: Achieving both high-fidelity visual rendering (via 3DGS and UE5) and complex physical simulation (via industrial solvers) in a unified environment.
• Scalability: Moving beyond "toy" isolated environments to complex, real-world scene compositions.
• Production Readiness: The pipeline targets offline, cinematic-quality production where simulation accuracy and visual fidelity are prioritized.

The authors acknowledge that the current system is not yet real-time and focuses on offline rendering. Future work is suggested for real-time interactive scenarios and improving object-level separation in complex captured scenes. The work does not claim to solve all physics challenges but specifically addresses the integration of 3DGS into existing, powerful heterogeneous physics ecosystems.