💻 computer science

EMPM: Embodied MPM for Modeling and Simulation of Deformable Objects

This paper introduces EMPM, a novel framework that combines differentiable Material Point Method simulation with multi-view RGB-D video reconstruction and online parameter optimization to create physically plausible, data-efficient, and adaptive models of deformable objects for robotic manipulation.

Original authors: Yunuo Chen, Yafei Hu, Lingfeng Sun, Tushar Kusnur, Laura Herlant, Chenfanfu Jiang

Published 2026-01-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yunuo Chen, Yafei Hu, Lingfeng Sun, Tushar Kusnur, Laura Herlant, Chenfanfu Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to handle squishy, messy things like dough, playdough, or a wet towel. The problem is that these objects don't behave like rigid boxes or balls; they stretch, tear, and change shape in complex ways. If you just guess how they move, the robot will likely drop the dough or rip the towel.

This paper introduces a new system called EMPM (Embodied MPM) that acts like a "super-smart digital twin" for these squishy objects. Here is how it works, broken down into simple concepts:

1. The "Digital Clay" (The Simulator)

Think of the old way of simulating soft objects like building a model out of springy balls connected by rubber bands (a "spring-mass" system). It's simple, but if you try to tear a piece of bread or squeeze a lump of clay, the rubber bands just bounce back or break in unrealistic ways.

The authors use a different tool called MPM (Material Point Method). Imagine instead of rubber bands, you have a cloud of tiny, invisible marbles floating inside the object. These marbles can slide past each other, stick together, or separate. This allows the computer to simulate complex things like:

Stretching: Like pulling taffy.
Squeezing: Like squishing playdough.
Tearing: Like ripping a piece of pita bread.

2. The "Mirror World" (Real-to-Sim-to-Real)

The system doesn't just guess the rules of physics; it learns them by watching the real world.

The Setup: They use multiple cameras (like a team of security guards) to film a human or robot manipulating an object.
The Learning: The computer watches the video and builds a 3D map of the object. It then runs its "Digital Clay" simulation in the background.
The Correction: If the simulation says "the dough should stretch this much," but the video shows "it stretched more," the computer realizes its guess about the dough's "squishiness" (stiffness) was wrong. It automatically tweaks the numbers in its simulation until the Digital Clay moves exactly like the Real Dough.

3. The "Live Tuning" (Online Optimization)

Usually, you have to do all this learning before the robot starts working. But EMPM is special because it can learn while it works.

Imagine you are playing a video game where the controller is slightly off. You adjust your grip, and the game instantly realizes, "Oh, the character is moving too fast," and slows down the physics engine in real-time to match your actual movements.
EMPM does this with robots. As the robot moves, the system constantly checks the video feed. If the object starts behaving differently than expected (maybe the dough got warmer and softer), the system updates its internal "recipe" for that object on the fly.

4. What They Proved

The researchers tested this on a wide variety of "squishy" things:

Elastic things: A rope, a piece of cloth, and a bouncy toy (things that snap back).
Plastic things: Playdough, bread dough, and pita bread (things that stay changed or tear).

They compared their "Digital Clay" (MPM) against the old "Rubber Band" (Spring-Mass) method. The results showed that:

The old method failed miserably when trying to tear bread or squeeze playdough (it just bounced or broke unrealistically).
Their new method handled all these scenarios perfectly, creating a simulation that looked and moved almost exactly like the real object.

In a Nutshell

This paper presents a system that lets robots "feel" and predict how soft objects will move by creating a live, self-correcting 3D simulation. It learns the physical properties of an object (like how stiff or stretchy it is) just by watching it move, allowing the robot to manipulate complex, messy materials with much higher accuracy than before.

Technical Summary: EMPM – Embodied MPM for Modeling and Simulation of Deformable Objects

1. Problem Statement

Accurately modeling deformable objects—specifically continuum materials like elastic and elastoplastic substances—remains a fundamental challenge in 3D vision, computer graphics, and robotics. Existing approaches face significant limitations:

Oversimplification: Traditional methods often rely on spring-mass systems or particle-based models that fail to capture complex dynamics such as large deformations, fracture, tearing, and permanent shape changes (plasticity).
Data Inefficiency: Learning-based intuitive physics models require large training datasets and often generalize poorly to novel objects.
Lack of Differentiability: Many simulation frameworks lack the ability to optimize physical parameters directly from visual observations, hindering the creation of accurate "digital twins" for robotic manipulation.

The core challenge addressed by this work is developing a framework that is physically plausible, generalizable, and data-efficient, capable of reconstructing geometry, appearance, and dynamics from multi-view RGB-D videos to enable differentiable physics simulation.

2. Methodology: EMPM Framework

The authors propose EMPM (Embodied MPM), a real-to-sim-to-real pipeline that integrates geometry reconstruction, appearance rendering, and a differentiable Material Point Method (MPM) simulator. The system operates in two modes: offline identification and online adaptive identification.

2.1. Core Components

Differentiable MPM Simulator: The foundation of the system is an MPM engine that couples continuum balance laws with a hybrid particle-grid discretization. This allows for the simulation of large deformations, contact, and elastoplastic flow. The simulator is action-conditioned, meaning it takes controller actions (e.g., robot gripper or hand motions) as boundary conditions.
Geometry and Appearance Reconstruction:
- Input: Multi-view RGB-D videos.
- Process: The system uses Grounded SAM2 to segment objects and controllers. It fuses depth data to reconstruct a 3D point cloud.
- Rendering: A 3D Gaussian Splatting (3DGS) model is built for high-fidelity appearance rendering. The motion of Gaussian centers is interpolated with MPM particle nodes using Linear Blend Skinning (LBS).
Parameter Optimization: The system optimizes material parameters (Young's modulus $E$ , Poisson's ratio $\nu$ , density $\rho$ , and plastic yield stress $y$ ) by minimizing the discrepancy between predicted simulation rollouts and real-world observations.

2.2. Offline Simulation and Optimization

In the offline setting, the system processes pre-collected RGB-D data:

Data Processing: 3D point clouds are lifted from multi-view images, and temporal correspondences are established using off-the-shelf point tracking models.
Objective Function: The loss function $L_{offline}$ $L_{o f f l in e}$ minimizes the difference between the simulated particle states ( $\hat{X}$ $\hat{X}$ ) and observed states ( $\tilde{X}$ $\tilde{X}$ ):
- Chamfer Distance: Measures the geometric difference between point clouds.
- Tracking Loss: Minimizes the $L_2$ distance between individual tracked particles to align motions.
Optimization: Parameters are updated using gradient-based optimization (AdamW) via automatic differentiation (Warp framework) or zero-order optimization (CMA-ES).

2.3. Online Adaptive Identification

The system extends to real-time interaction with streaming video:

Streaming Supervision: Initial point clouds are generated from the start of the stream. Subsequent frames are segmented and back-projected to update the 3D observation.
Constraint Handling: Due to occlusion degrading point tracking reliability over time, online optimization does not rely on point tracking. Instead, it assumes quasi-static equilibrium (e.g., when an object is pinned).
Loss Function: The online loss $L_{online}$ $L_{o n l in e}$ combines:
- 3D Chamfer Distance: Aligns point clouds.
- 2D Segmentation Mask Loss: Regularizes the alignment of simulated particles with the 2D segmentation mask of the observed object.
Adaptation: Parameters are updated step-by-step to correct the digital twin's behavior based on live sensory feedback.

3. Key Contributions

Real-to-Sim-to-Real System: A novel framework based on differentiable MPM physics simulation that jointly performs parameter optimization, geometry reconstruction, and appearance rendering from real sensory observations.
Online Parameter Optimization: An adaptive scheme that updates material parameters during live interactions, enabling robust, physics-aware object representations for downstream robotic control.
Broad Material Handling: Demonstration of the method on a wide range of deformable objects, including both elastic (rope, cloth, soft toys) and elastoplastic (plasticine, dough, pita bread) materials, which are challenging for existing baselines.

4. Experimental Results

The authors evaluated EMPM on six types of objects involving manipulations like squeezing, stretching, tearing, and poking.

Comparison with Baselines:
- PhysTwin [8] (Spring-Mass): EMPM significantly outperformed PhysTwin, particularly in modeling fracture (e.g., tearing pita bread) and plastic deformation (e.g., squeezing plasticine). Spring-mass models failed to capture permanent shape changes and fracture dynamics.
- PGND [10] (Learning-based): EMPD achieved better alignment in both geometric and visual metrics compared to the neural dynamic baseline.
Quantitative Metrics:
- Geometric Accuracy: EMPM achieved lower Chamfer distances and tracking errors across both elastic and elastoplastic categories.
- Visual Quality: EMPM demonstrated superior performance in IoU, PSNR, SSIM, and LPIPS metrics for photorealistic rendering.
- Efficiency: While PGND had faster inference times, EMPM was significantly more efficient in training time compared to both PGND and PhysTwin due to batch optimization over entire training frames.
Online Optimization: Experiments with a bimanual Franka robot showed that online correction progressively improved the alignment between the simulated digital twin and real observations, reducing both 2D mask and 3D distance errors.

5. Significance and Claims

The paper claims that EMPM opens new possibilities for the robotic manipulation of complex deformables. By providing a differentiable, physics-aware simulation that can be optimized online from visual data, the framework enables:

The construction of photo- and physics-realistic digital twins.
Physics-aware robotic manipulation tasks, where the robot can predict object responses to hypothetical controller inputs (what-if analysis) and plan trajectories accordingly.
A robust alternative to oversimplified spring-mass models and data-hungry learning-based approaches, particularly for objects exhibiting non-elastic behaviors like plasticity and fracture.

The authors acknowledge limitations regarding the reliability of point tracking under heavy occlusion and large deformations, noting that future work could benefit from emerging tracking models to further enhance online optimization.