Technical Summary: DeformMaster

Problem Statement

World models for deformable objects must recover not only geometry and appearance but also underlying physical dynamics, interaction grounding, and material behavior. Learning such models from real-world videos is challenging because deformable objects (linear, planar, and volumetric) evolve in high-dimensional state spaces governed by distributed strain, complex material responses, self-contact, and external forces.

Existing approaches face significant limitations:

• Physics-based reconstruction (e.g., PhysTwin, Spring-Gaus) often relies on idealized physics models where parameter identification struggles to account for real-world phenomena beyond model assumptions.
• Generative models (e.g., video diffusion) primarily imagine visual outcomes but often lack reliable understanding of action-conditioned dynamics, making them difficult to control via explicit interactions and prone to hallucinated dynamics.
• Purely learned dynamics (e.g., particle-graph networks) suffer from heavy data dependency, remain tied to training distributions, and drift significantly during long-horizon rollouts under novel actions.

The core challenge is to build a model that infers physical states from visual observations, rolls them forward under new interactions, and renders high-fidelity dynamics while remaining stable, generalizable, and physically grounded.

Methodology: DeformMaster

DeformMaster is a video-derived interactive physics-neural world model that unifies dynamics and appearance within a single framework. It transforms real interaction videos into an online, queryable representation supporting novel action rollouts, material parameter variations, and dynamic novel-view synthesis. The system comprises four key components:

1. Physics-Neural Particle-Grid Dynamics (PNPGD)

To balance the stability of physics priors with the flexibility to model real-world mismatches, DeformMaster decomposes the dynamics operator $F_{\theta,\phi}$ into a physics block $P_\theta$ and a neural residual $R_\phi$:
$$F_{\theta,\phi} = P_\theta \oplus R_\phi$$

• Physics Block ($P_\theta$): Uses differentiable Material Point Method (MPM) to advance the state over one frame based on material parameters $\theta$. This provides a structured, stable prior.
• Neural Residual ($R_\phi$): A particle-grid neural network that predicts a bounded velocity correction $\Delta v_p$ based on the post-MPM state and a short kinematic history. This residual absorbs unmodeled effects (e.g., friction, complex contact) without acting as a free state predictor, ensuring the model remains a perturbation of the physics simulation rather than a complete replacement.

2. Distributed Compliant Actuators (DCA)

Real-world hand tracks derived from vision are noisy and sparse. Hard pointwise constraints often lead to velocity spikes or localized deformations that fail to drive bulk motion. DCA addresses this by:

• Compliance: Converting hard constraints into compliant actuator-particle couplings (using stiffness $k_p$ and damping $k_d$) to absorb high-frequency noise.
• Distribution: Spreading actuation over a finite contact patch (local neighborhood) rather than a single point, enabling effective force transmission through soft continua.

3. Mixture of Constitutive Experts (MoCE)

Real deformable objects rarely conform to a single idealized constitutive law. MoCE represents the first Piola–Kirchhoff stress as a spatially varying mixture of canonical experts (Neo-Hookean, Corotated, St. Venant–Kirchhoff):
$$P_{mix}(F_p) = \sum_k w_{k,p} P_k(F_p; E_p, \nu_p)$$
The mixture weights $w_{k,p}$ are spatially varying, allowing the model to capture heterogeneous material responses across different regions of the object while remaining grounded in analytic constitutive laws.

4. Physics-Grounded 4D Appearance

To render high-fidelity appearance without re-optimizing a dynamic model for every action, DeformMaster drives a Gaussian Splatting representation using the predicted material particle trajectory. A bridge function $B_\psi$ deforms Gaussians using Linear Blend Skinning (LBS) incrementally based on particle motion. This ensures the rendered appearance remains consistent with the underlying physical evolution.

Training Scheme

The model is trained with a combined objective minimizing dynamics errors (track alignment, shape Chamfer distance, local length preservation) and appearance errors (RGB photometric loss). Training proceeds in stages:

1. Stages 1–2: Alternating optimization between dynamics training (learning material fields and neural residuals) and Distributed Compliant Actuator (DCA) gain selection using CMA-ES.
2. Stage 3: Optimization of Gaussian splats followed by RGB-guided refinement of the dynamics branch, where photometric gradients flow back to the particle motion and neural residual.

Key Contributions

1. PNPGD: A hybrid transition model that augments differentiable physics with a neural residual, preserving physics-guided rollouts while compensating for unmodeled real-world effects.
2. DCA: A mechanism to transform noisy, sparse hand tracks into compliant, spatially distributed actuation, stabilizing hand-continuum interaction.
3. MoCE: A constitutive modeling approach that blends canonical material laws with spatially varying weights to capture heterogeneous material responses.
4. DeformMaster System: A unified video-derived world model that pairs interactive physics-neural dynamics with physics-grounded high-fidelity 4D appearance, supporting online interaction.

Experimental Results

DeformMaster was evaluated on 20 real-world deformable-object sequences from the PhysTwin dataset, covering linear (ropes), planar (cloths/packages), and volumetric (soft toys) objects.

• Overall Performance: DeformMaster achieved the strongest overall performance, outperforming baselines (PhysTwin, Spring-Gaus, GS-Dynamics) in both future dynamics (IoU, Chamfer distance) and rendered appearance (PSNR, SSIM, LPIPS).
• Comparison with PhysTwin: While maintaining comparable track error, DeformMaster improved Chamfer distance (0.011 vs. 0.012) and IoU (0.748 vs. 0.734).
• Comparison with Learned Dynamics: The method significantly reduced Chamfer distance (by >3$\times$) and track error (by >2.9$\times$) compared to purely learned Gaussian dynamics baselines.
• Generalization: The model successfully supported rollouts under novel actions, material parameter variations (e.g., scaling material fields to induce fracture), and novel view synthesis.
• Ablation Studies:
  • Removing the neural residual degraded performance, confirming the need to bridge idealized physics and real observations.
  • Replacing DCA with rigid constraints or single-point actuators caused divergence or reduced accuracy, validating the need for compliance and distribution.
  • Replacing MoCE with a single expert worsened results, demonstrating the necessity of spatially varying material modeling.
  • RGB-guided refinement in Stage 3 provided consistent gains in both dynamics and appearance metrics.

Significance and Claims

The paper claims that DeformMaster represents a significant step toward robust, interactive world models for deformable objects. By jointly recovering physical dynamics, interaction grounding, and material behavior, the model overcomes the limitations of purely generative approaches (which lack action controllability) and purely physics-based approaches (which struggle with real-world complexity).

The authors emphasize that DeformMaster enables:

• Stable long-horizon rollouts under novel actions.
• Robust interaction despite noisy visual observations.
• Heterogeneous material modeling without assuming a single global constitutive law.
• High-fidelity 4D appearance driven directly by physical evolution.

The work positions itself as a foundation for downstream embodied applications, such as model-predictive-control-based robotic manipulation and synthetic data generation, while acknowledging limitations regarding very thin planar objects (where spring-mass systems may still hold an inductive bias) and runtime dependencies on simulation substeps.