Resonance4D: Frequency-Domain Motion Supervision for Preset-Free Physical Parameter Learning in 4D Dynamic Physical Scene Simulation

1. Problem Statement

The paper addresses the challenge of generating physically consistent 4D dynamic simulations from static 3D reconstructions (specifically 3D Gaussian Splatting). Two primary bottlenecks limit existing methods:

• High Computational Cost of Supervision: Current state-of-the-art methods rely on heavy online video diffusion models or optical flow networks to enforce temporal consistency. These auxiliary models consume excessive GPU memory (often >35 GB) and computational resources, making them unsuitable for consumer-grade hardware or long-horizon simulations.
• Incomplete Physical Modeling: Existing approaches often simplify inverse physical modeling by optimizing only a subset of material parameters (e.g., just Young's modulus) while fixing others as constants. Furthermore, they typically use coarse object-level parameterization, failing to capture the spatially heterogeneous material properties (e.g., different stiffness in leaves vs. stems) found in real-world non-rigid objects.

2. Methodology: Resonance4D

The authors propose Resonance4D, a unified framework coupling 3D Gaussian Splatting (3DGS) with the Material Point Method (MPM). The core philosophy is to replace heavy online priors with lightweight, physically expressive supervision and to enable full-parameter recovery through part-level optimization.

The framework consists of three key components:

A. Dual-Domain Motion Supervision (DMS)

Instead of relying on external video generation models, Resonance4D enforces motion consistency through two complementary domains:

1. Spatial Domain (Structural Consistency): Uses standard frame-wise reconstruction loss (SSIM) to ensure the rendered video matches the reference video in appearance and structural deformation.
2. Frequency Domain (Spectral Consistency): To capture oscillatory motion and global dynamic patterns that spatial losses often miss, the method analyzes frame differences in the spatiotemporal frequency domain.
   • It extracts local patches from frame-difference volumes and applies a 3D Fourier Transform.
   • It minimizes the difference in spectral magnitude (energy distribution) and phase consistency between the simulated and reference videos.
   • Benefit: This captures periodic motions (like swaying) and high-frequency details without the need for large diffusion models, significantly reducing memory overhead.

B. Simulation-Driven Initialization

Joint optimization of multiple coupled physical parameters is highly sensitive to initialization. To avoid local optima:

• The system performs a coarse search using Latin Hypercube Sampling (LHS) in the log-space of physical parameters (e.g., Young's modulus, density).
• It runs short forward MPM simulations for candidate parameter sets.
• The candidate producing the video most similar to the reference (via lightweight metrics like MS-SSIM) is selected as the initialization point for the differentiable optimization.

C. Part-Level Joint Physical Parameter Optimization

To handle complex, heterogeneous materials:

• Zero-Shot Segmentation: The system uses vision-language models (DINOv3 + SAM) to automatically segment the 3D Gaussians into semantic parts (e.g., "leaves," "stem") without manual annotation.
• Part-Level Parameterization: Instead of optimizing per particle (unstable) or per object (too coarse), parameters are optimized at the part level.
• Full-Parameter Recovery: The framework jointly optimizes a comprehensive set of constitutive parameters and material attributes, including:
  • Young's Modulus
  • Poisson's Ratio
  • Yield Stress
  • Plastic Viscosity
  • Friction Angle
  • Density (often fixed in other methods, but optimized here).
• Optimization is performed in the log-domain to ensure positivity and stability.

3. Key Contributions

1. Lightweight Supervision: Demonstrates that effective dynamic supervision does not require heavy online diffusion models. The proposed Dual-Domain Motion Supervision (DMS) combines spatial and spectral constraints to achieve high fidelity with significantly lower memory usage.
2. Preset-Free Full-Parameter Learning: Introduces a strategy for part-level joint optimization of all major physical parameters (including density), overcoming the limitations of partial parameter tuning and coarse object-level modeling.
3. Efficient Initialization: Develops a simulation-driven initialization strategy that uses rapid candidate screening to provide stable starting points for complex inverse problems.
4. Consumer-Grade Feasibility: The framework reduces peak GPU memory usage from >35 GB (in prior methods) to ~20 GB, enabling high-fidelity physics-driven 4D simulation on a single consumer-grade GPU.

4. Experimental Results

The method was evaluated on both real-world (PhysDreamer dataset) and synthetic (PAC-NeRF dataset) benchmarks.

• Quantitative Performance:
  • Real-World (PhysDreamer): Resonance4D achieved the best scores in MS-SSIM (0.7323), PSNR (18.56), and LPIPS (0.2857) across four diverse scenes (Alocasia, Carnations, Hat, Telephone).
  • Memory Efficiency: It reduced peak GPU memory usage by nearly 40% compared to the best baseline (DreamPhysics), dropping from ~33 GB to ~21 GB.
  • Synthetic (PAC-NeRF): Achieved the lowest Chamfer Distance (0.0063) and Hausdorff Distance (0.0914), indicating superior geometric accuracy in dynamic reconstruction.
• Qualitative Analysis:
  • Visual comparisons show Resonance4D better preserves oscillatory motion (e.g., swaying flowers) and frequency characteristics compared to baselines, which often suffer from temporal discontinuities or overly rigid/soft deformations.
  • The part-level density optimization resulted in more natural motion (e.g., distinct swaying behaviors for leaves vs. stems) compared to uniform density settings.
• Ablation Studies:
  • Confirmed that Frequency-Domain supervision is crucial for capturing periodic motion.
  • Showed that Simulation-driven initialization significantly improves convergence over random or GPT-based initialization.
  • Validated that Part-level optimization outperforms both particle-level and object-level approaches.
  • Proved that optimizing density alongside constitutive parameters is essential for realistic dynamic reconstruction.

5. Significance

Resonance4D represents a significant step forward in physics-driven 4D scene generation by decoupling high-fidelity simulation from the prohibitive costs of large-scale video priors. By shifting the focus to frequency-domain analysis and part-level physical inversion, the authors demonstrate that lightweight, interpretable constraints can yield results comparable to or better than heavy generative models. This makes high-quality, interactive 4D physical simulation accessible on consumer hardware, opening new possibilities for robotics, immersive content creation, and world modeling without requiring massive computational resources.