Gen2Physics: Grounding Generated 3D Meshes in Physics via Multi-View Material Decomposition

In the digital world, computers have become remarkably skilled at drawing three-dimensional objects. Given a single photograph or a few words, modern artificial intelligence can conjure up a chair, a robot, or a toy with stunning visual detail. These digital creations look real, but they are hollow shells. To a computer, a wooden chair with metal legs is just one continuous surface, a single object without internal structure. This creates a significant problem for anyone trying to use these images in video games, robotics, or physics simulations. Real-world objects behave according to their materials; a heavy metal head on a light wooden body will tip and rotate differently than a uniformly dense block. Without knowing which part is made of what, a computer cannot simulate how an object will fall, bounce, or roll. For decades, the gap between how an object looks and how it behaves has been a major hurdle, leaving digital creators with beautiful but static assets that cannot interact with the physical laws of a simulation.

A team of researchers has developed a new system called Gen2Physics to bridge this gap. Their goal was to take a static, generated 3D mesh and automatically figure out what it is made of, breaking it down into separate, simulation-ready pieces. Instead of trying to build the object from scratch with physical properties in mind, their method works after the object has already been created. It acts like a digital detective that looks at the object from many different angles, identifies the material of every tiny patch on its surface, and then reconstructs the object as a collection of solid parts, each with its own weight and density. The result is a digital asset that not only looks correct but also behaves correctly when dropped or pushed in a physics engine.

The process begins by treating the 3D object like a sculpture that needs to be examined from every side. The system renders the object from eleven different viewpoints, much like walking around a statue to see it from all sides. For each of these views, a specialized computer vision model analyzes the image to guess the material of every pixel. It looks for clues like shininess, texture, and color to decide if a surface is wood, metal, glass, or fabric. However, looking at an object from just one angle can be misleading. A reflection might make a wooden leg look like metal, or a shadow might hide a plastic handle. To solve this, the system projects all eleven views back onto the 3D model and uses a voting system. If ten out of eleven views agree that a specific face of the object is metal, the system accepts that as the truth. This step ensures that the final map of materials is consistent and free of the confusion that comes from a single perspective.

Even with a consistent map, the computer can still make mistakes based on what it sees. A thin, shiny leg might look like metal to a camera, but if the object is a chair, common sense suggests it is likely wood. To fix these logical errors, the system employs a large language model, a type of artificial intelligence trained to understand context. This model looks at the object as a whole, considering its shape and function, and corrects the material labels. It might decide that a part labeled "metal" is actually "wood" because it is part of a chair leg, or that a "plastic" part is actually "glass" because it is a transparent vase. This step adds a layer of reasoning that pure image analysis cannot achieve, ensuring the materials make sense in the real world.

Once the materials are correctly identified, the system faces a final challenge: turning these flat surface patches into solid, watertight volumes. A simulation engine cannot work with a hollow shell; it needs to know the inside of the object to calculate how heavy it is and how it will spin. The system fills in the missing geometry, creating complete 3D shapes for each material part. It then asks the language model to guess whether each part is solid all the way through or hollow, like a pipe or a shell. If a part is hollow, the system estimates the thickness of the walls. With these details, it can calculate the exact mass and weight distribution of the entire object. This allows the computer to know, for instance, that the heavy metal head of a robot is at the top, which will cause it to tip over if pushed, rather than spinning perfectly around its center.

The researchers tested their method on a large collection of 3D models and compared it to previous techniques. They found that their approach was far more accurate at identifying materials, correctly labeling nearly half of the parts in their test set, which was more than double the success rate of earlier methods. When they simulated the objects falling and colliding, the results were striking. In one test, a robot head made of heavy metal and a wooden body was dropped. The older methods, which treated the whole object as having the same density, caused it to rotate around its geometric center in an unnatural way. The new system, however, correctly identified the heavy metal head, calculated the true center of mass, and simulated a fall that matched real-world physics, with the heavy head pulling the object down and causing it to rotate realistically.

This work represents a significant step forward in making digital content usable for interactive applications. By automatically decomposing complex 3D shapes into their material components and assigning them realistic physical properties, the system turns static images into dynamic, interactive objects. The researchers note that while their method is highly effective, it still relies on external tools to fill in the missing geometry and depends on the quality of the initial 3D model. They also point out that their current tests are limited to objects where the materials are visible on the surface, meaning objects with complex internal structures that cannot be seen from the outside remain a challenge. Nevertheless, the ability to generate assets that are both visually faithful and physically plausible opens the door for more realistic simulations in robotics, gaming, and engineering, bringing the digital and physical worlds closer together.