🔬 applied physics

SoMA: A Real-to-Sim Neural Simulator for Robotic Soft-body Manipulation

This paper introduces SoMA, a 3D Gaussian Splat-based neural simulator that unifies deformable dynamics, environmental forces, and robot actions in a latent space to achieve accurate, stable, and generalizable real-to-sim soft-body manipulation without relying on predefined physical models.

Original authors: Mu Huang, Hui Wang, Kerui Ren, Linning Xu, Yunsong Zhou, Mulin Yu, Bo Dai, Jiangmiao Pang

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Mu Huang, Hui Wang, Kerui Ren, Linning Xu, Yunsong Zhou, Mulin Yu, Bo Dai, Jiangmiao Pang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to fold a laundry basket full of t-shirts. It sounds simple, but for a machine, a piece of cloth is a nightmare. Unlike a rigid box, a shirt has no fixed shape; it twists, stretches, and hides parts of itself from the robot's eyes. To learn how to handle these floppy objects, robots usually need to practice millions of times. But practicing in the real world is slow, expensive, and risks breaking things. So, scientists build "simulators"—digital playgrounds where robots can practice. The problem is that most digital playgrounds are either too stiff (they use strict math rules that don't match real life) or too dreamy (they guess what happens next but get the physics wrong). The goal of this research is to build a simulator that is both accurate enough to be trusted and flexible enough to handle the messy reality of squishy objects.

Enter SoMA, a new kind of "neural simulator" designed specifically for soft-body manipulation. Think of SoMA not as a calculator crunching physics equations, but as a super-smart student who learns by watching videos. Instead of being told how a shirt should move, SoMA watches a robot actually move a shirt in the real world, records the video, and then figures out the hidden rules of motion on its own. It uses a clever trick called "Gaussian Splatting," which is like turning the object into a cloud of millions of tiny, glowing, 3D paint splatters. Instead of trying to calculate the physics of every single thread, SoMA learns how these splatters dance and deform when the robot pushes them.

The paper shows that SoMA can take a video of a robot interacting with objects like ropes, dolls, and shirts, and then recreate that interaction in a digital world with surprising accuracy. When tested, SoMA didn't just copy the video; it learned the underlying "feel" of the object. If you asked it to simulate a new way of folding a shirt that it had never seen before, it could still do it correctly, whereas older methods would often get confused or make the object look like it was melting. The authors found that by combining what the robot is doing (its arm movements) with what the object looks like, they could create a simulation that is 20% more accurate than previous top methods. This means we might soon have digital twins of real-world tasks that are reliable enough to train robots to do complex chores without needing to break a single real-world shirt first.

The Magic of "SoMA"

The Problem: The "Squishy" Gap
Robots are great at moving boxes. They are terrible at moving laundry. Why? Because a box is rigid; if you push it, it moves. A shirt is "soft-body," meaning it bends, folds, and hides itself. To teach a robot to fold a shirt, you need a simulator that understands how cloth behaves.

Old Way 1 (The Rulebook): Scientists used to write strict physics rules (like gravity and friction) into the computer. But getting these rules perfect for a real shirt is nearly impossible. If the numbers are slightly off, the digital shirt behaves like a rock or a jellyfish, not a shirt.
Old Way 2 (The Dreamer): Other methods tried to learn from data by just guessing the next frame of a video. These are good at looking real but often fail when the robot does something new. They might make the shirt float or pass through the table because they don't truly understand the "push" from the robot.

The Solution: Learning from "Splats"
SoMA introduces a new way to represent the world. Instead of a mesh or a solid block, it represents the object as a collection of Gaussian Splatting (GS) points. Imagine the shirt isn't made of fabric, but of millions of tiny, fuzzy, glowing dots floating in 3D space.

How it works: When the robot grabs the shirt, it doesn't just move the dots; it applies "forces" to them. SoMA learns how these forces travel through the cloud of dots.
The Secret Sauce: SoMA doesn't just watch the video; it watches the robot's joints too. It knows exactly how the robot's arm moved. It connects the robot's action directly to the movement of the dots. This is called "action-conditioned." It's like the simulator knows, "When the robot's elbow bends this way, the shirt's left corner must lift that high."

The Training: A Two-Step Dance
Training a simulator to handle long tasks (like folding a whole shirt) is hard because small mistakes add up quickly. If the simulator is off by a millimeter in the first second, by the tenth second, the shirt might be floating in the air.

Step 1 (The Big Picture): SoMA first learns the big, slow movements by looking at the video with large gaps between frames. It learns the general flow of the shirt.
Step 2 (The Details): Then, it zooms in to learn the tiny, fast details, like how the fabric wrinkles when it hits the table.
The "Blended" Lesson: Since the robot's hand often blocks the view of the shirt (occlusion), the simulator can't see everything. SoMA uses a special trick: it only learns from the parts of the video it can see, but it uses "physics rules" (like mass conservation) to guess what's happening in the hidden parts. This keeps the shirt from falling apart when it's hidden behind the robot's hand.

The Results: Better Than Before
The researchers tested SoMA on real-world data involving ropes, dolls, and t-shirts.

Accuracy: In tests, SoMA achieved a PSNR of 33.51 for resimulation (copying what it saw) and 32.89 for generalization (predicting new actions). This is a 20% improvement over the previous best methods.
Stability: While other simulators would eventually make the object look weird or unstable after a few seconds, SoMA could simulate long, complex tasks (like folding a t-shirt) without the object collapsing or drifting away.
Generalization: When asked to perform a manipulation it had never seen before, SoMA didn't just guess randomly; it used the robot's actions to guide the simulation, producing realistic results.

Why It Matters
This isn't just about making pretty videos. By creating a simulator that is this accurate and stable, we can train robots in the digital world to do complex, squishy tasks. Once the robot learns in SoMA, it can transfer that skill to the real world with much less trial and error. It bridges the gap between the messy reality of soft objects and the clean logic of computer code, potentially helping robots fold laundry, handle food, or assist in delicate manufacturing tasks much faster than before. The paper suggests that this approach is a significant step toward making robots that can truly understand and interact with the soft, flexible world we live in.

Technical Summary: SoMA - A Real-to-Sim Neural Simulator for Robotic Soft-body Manipulation

1. Problem Definition

Simulating deformable objects under rich robot interactions remains a fundamental challenge for real-to-sim (R2S) robot manipulation. Existing approaches face a trade-off between physical fidelity and interaction consistency:

Physics-based simulators (e.g., FEM, MPM) ensure long-horizon consistency but rely on predefined physical parameters that are difficult to infer from visual data, leading to R2S bias.
Differentiable simulators optimize limited parameters but remain constrained by simplified physical assumptions.
Neural dynamics and 4D reconstruction methods learn motion directly from data but primarily focus on reproducing observed trajectories. They often lack support for robot-conditioned interaction and fail to generalize beyond the training distribution, particularly under partial occlusion and complex contact patterns.

The core challenge is to develop a simulator that faithfully captures deformable object geometry and dynamics from visual observations while remaining stable and coherent under sustained, action-conditioned robot-object interaction.

2. Methodology: SoMA Framework

SoMA (Simulator of Manipulation for soft-body interaction) is a 3D Gaussian Splat (GS) neural simulator that operates directly on learned Gaussian splat representations. It unifies deformable dynamics, environmental forces, and robot joint actions in a single latent neural space without predefined physical models.

2.1. Scene Initialization via R2S Mapping

To bridge the gap between heterogeneous coordinate systems (visual reconstruction, robot kinematics, and physical frames), SoMA establishes a unified simulation space:

Reconstruction: Multi-view RGB images are used to estimate camera poses and reconstruct the deformable object as a set of Gaussian splats ( $G_0$ ) in the camera coordinate system.
Robot Conditioning: A global scale factor and rigid transformation are estimated to align the reconstructed geometry with the robot's base frame. Robot joint configurations ( $q_t$ ) are converted to end-effector poses via forward kinematics and mapped to the simulation space.
Physical Reference: Gravity direction is resolved by fitting the supporting table plane, ensuring the simulation aligns with real-world physical constraints.

2.2. Force-Driven GS Dynamics Modeling

Unlike state-based regression models, SoMA models interactions as explicit forces acting directly on Gaussian splats:

Environment Forces: Modeled as external forces including gravity and support forces for splats near the supporting surface.
Robot Forces: Modeled via an interaction graph connecting robot control points and GS nodes. A graph-based neural module predicts robot-induced forces based on the current state, neighboring robot points, and gripper state.
Hierarchical Propagation: The total interaction force is hierarchically propagated through a multi-level graph structure. A neural dynamics model ( $\psi_\theta$ ) predicts linear velocity and rotation for each node/cluster conditioned on historical states and aggregated forces, enabling force-driven rather than state-only dynamics.

2.3. Multi-Resolution Training Strategy

To address error accumulation and instability in long-horizon simulation, SoMA employs a two-stage training approach:

Temporal: A coarse-to-fine strategy where the model first learns global motion patterns with large temporal strides ( $k \cdot dt$ ), followed by fine-grained dynamics learning at the original resolution ($dt$) using random subsequences.
Spatial: GS geometry is reconstructed from super-resolution images to preserve details, while dynamics training occurs at original resolution to reduce computational cost.

2.4. Blended Supervision

To handle partial observability and occlusion during interaction, SoMA uses a blended supervision scheme:

Occlusion-Aware Image Supervision: Image reconstruction loss is applied only to visible object regions (masked by binary object masks) to prevent spurious gradients from occluded areas.
Physics Consistency Regularization: A self-supervised constraint enforces mass conservation between adjacent hierarchy levels. This regularizes unobserved dynamics, preventing physically inconsistent motion and drift without requiring ground-truth forces.

3. Key Contributions

R2S Neural Simulation Paradigm: A novel framework for soft-body manipulation that supports long-horizon, interaction-consistent simulation directly from RGB observations and robot actions.
Feasibility Mechanisms:
- Robot-Conditioned Alignment: Anchors learned dynamics to real robot kinematics for causal interaction.
- Force-Driven Dynamics: A Gaussian-splat dynamics model that handles contact and occlusion via explicit force modeling.
- Stable Training: Multi-resolution training and blended supervision for stable long-horizon performance.
Empirical Validation: Extensive evaluation on public benchmarks and a new real-world dataset (rope, doll, cloth, T-shirt) demonstrating state-of-the-art performance in RGB and depth metrics.

4. Experimental Results

The authors evaluated SoMA on resimulation (training trajectories) and generalization (unseen manipulation sequences) across four deformable objects.

Quantitative Performance: SoMA outperformed baselines (PhysTwin and GausSim) across all metrics.
- Resimulation: Achieved a PSNR of 33.51 (vs. 28.77 for PhysTwin) and an Abs Rel depth error of 0.089 (vs. 0.102).
- Generalization: Achieved a PSNR of 32.89 and Abs Rel of 0.112, showing a 20% improvement over baselines in generalization accuracy.
Complex Tasks: In the challenging T-shirt folding task (involving large deformations and self-contact), SoMA maintained coherent geometry and realistic dynamics, whereas PhysTwin exhibited severe artifacts and structural collapse.
Ablation Studies: Removing blended supervision (Img-only) or multi-resolution training (w/o MRT) resulted in significant performance drops, confirming the necessity of physics-inspired constraints and multi-scale training for generalization and stability.

5. Significance and Claims

The paper claims that SoMA bridges the gap between physics-based and data-driven simulation by explicitly modeling robot-object interactions within a unified Gaussian splat representation. Its primary significance lies in:

Eliminating Predefined Physics: It learns dynamics end-to-end from data without relying on manually specified physical parameters, addressing the difficulty of parameter identification in R2S scenarios.
Stable Long-Horizon Interaction: It enables stable simulation of complex, long-horizon tasks (e.g., cloth folding) where existing neural simulators typically drift or collapse.
Practical R2S Backend: By reducing the reality gap, SoMA serves as a foundation for data synthesis, augmentation, and policy learning for embodied robot manipulation, potentially reducing the cost and risk of real-world data collection.

The authors position SoMA as a distinct approach that balances physical consistency, controllability, and fidelity to real-world observations, specifically tailored for robot-manipulated, occlusion-heavy embodied scenarios.