DeformSmith: Physics Harness-Guided Hierarchical Generation of Deformable Assets for Robot Manipulation

In the virtual worlds where robots learn to move, the objects they interact with are often too perfect. A digital apple is a rigid sphere that never dents; a digital towel is a flat sheet that never folds. For a robot to learn how to grasp, lift, or carry something in a simulation, that object must behave like the real thing. It must squish when squeezed, drape when dropped, and hold its shape when lifted. Creating these digital objects from scratch is difficult because the visual appearance of an object—what it looks like—is only half the story. The other half is its hidden physical nature: how heavy it is, how soft or stiff its material is, and how it reacts when touched. Without knowing these invisible properties, a robot might try to pick up a digital banana only to find it shatters like glass or slips through its fingers like water, simply because the computer did not know how to make it bend.

Researchers have long tried to generate these 3D objects using text descriptions or single photographs, but until now, the results have often been visually convincing yet physically broken. They might look like a plush toy, but in a simulation, they would collapse under their own weight or fail to react to a robot's grip. The challenge lies in the fact that a picture or a sentence does not contain enough information to tell a computer exactly how an object should deform. To solve this, a team of researchers has developed a new system called DeformSmith. This framework does not just guess what an object should look like; it builds the object in layers, testing its physical behavior at every step, and uses a simulated robot to try and manipulate the object before the process is finished.

The core of this new approach is a step-by-step construction process that mimics how a human engineer might build a prototype, but with a computer doing the heavy lifting. The system starts with a simple description or a single photo and first builds the 3D shape of the object. Once the shape exists, the system assigns it a physical identity, deciding how much it weighs and how it interacts with the ground. Then, it adds the material properties, determining if the object is soft like a sponge or firm like a rubber ball. Crucially, the system does not just set these values and hope for the best. It runs a series of physical tests, dropping the object or pressing on it, to see if it behaves correctly. If the object collapses too easily or bounces in the wrong way, the system automatically adjusts its internal settings and tries again.

What makes this method unique is how it brings a robot into the loop. After the object passes the basic physical tests, a simulated robot arm attempts to pick it up, carry it, and set it down. This is where the system learns the most. The robot tries to grasp the object, and the system watches closely to see if the object holds its shape, if it slips, or if it deforms in a way that makes it impossible to move. If the robot fails, the system uses that failure as a clue. It might realize the object is too slippery, or that the grip was too weak, or that the material is too soft for the task. The system then goes back and refines the object's properties or changes how the robot moves, repeating the cycle until the robot can successfully complete the task. This creates a feedback loop where the object is constantly improved based on how well it works in a real-world scenario.

The researchers tested this system by creating dozens of different objects, from a rugby ball to a plush seal, using both text descriptions and single images. They compared their results against other advanced methods that try to do the same thing. In these comparisons, the objects created by DeformSmith were significantly more realistic. When dropped, they maintained their shape and bounced or squished in a way that looked natural, whereas objects from other systems often flattened into a pancake or fell apart. In blind tests where people judged which object looked and behaved better, the new system won the majority of the time. It was particularly successful at creating objects that could be successfully manipulated by a robot, with the success rate for picking up and moving objects jumping from less than half to more than two-thirds when the robot-guided refinement was used.

The system works by breaking the problem down into manageable stages, ensuring that the shape is correct before worrying about the weight, and the weight is correct before worrying about the material. This prevents the computer from getting confused by trying to fix everything at once. A central "harness" acts as a supervisor, keeping track of all the rules and ensuring that changes made at one stage do not break the work done at a previous stage. For example, if the system decides to make an object softer, it checks that this change does not make the object too heavy or cause it to sink into the floor. This careful, hierarchical approach allows the system to build complex, interactive objects that are ready for use in robot training simulations.

While the system is powerful, the researchers note that it currently works best with solid objects that can be squeezed or stretched, rather than liquids or granular materials like sand. The physical properties it infers are also estimates based on visual cues and simulation, meaning they would still need to be verified with real-world measurements if used for actual physical robots. However, the ability to generate a wide variety of physically credible objects from simple inputs opens a new door for robotics. Instead of manually modeling every single object a robot might encounter, engineers can now describe an object or show a picture, and the system will build a digital twin that is ready to be tested, manipulated, and learned from. This capability suggests a future where robots can learn to handle the messy, deformable world of everyday objects much faster, simply by practicing with a vast library of automatically generated, physically accurate digital assets.