Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents
This paper introduces Agentic Real2Sim, a vision-language agent framework that automates the labor-intensive conversion of real-world robotic interactions into runnable physical simulations by recovering geometries, inferring physical parameters, and assembling scene components, thereby enabling scalable policy learning and evaluation across diverse manipulation and motion domains.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine trying to teach a robot to do a new trick, like stacking cups or folding laundry. You could spend months filming real humans doing it, but then you'd have to manually rebuild every single cup, table, and hand movement inside a computer program just to let the robot practice. It's like trying to teach a video game character by hand-painting every single pixel of the world they live in. This is the "Real-to-Sim" problem: turning messy, real-world video into a clean, playable digital twin. Scientists have been trying to automate this, but it's usually a fragile process where one tiny mistake in measuring a table's height or a cup's weight breaks the whole simulation.
Enter the world of "Agentic" AI. Think of an AI agent not as a simple calculator, but as a tiny, super-smart project manager. Instead of just following a rigid list of instructions, an agent can look at a messy situation, decide what tools to use, check its own work, and fix mistakes on the fly. This paper, titled "Agentic Real2Sim," asks a big question: Can we use these smart AI project managers to automatically turn a video of a robot doing a task into a high-quality, playable digital version of that exact moment? The goal isn't just to make a pretty 3D picture; it's to make a physics-accurate world where the robot can try the task again, learn from its mistakes, and get better, all without a human needing to tweak every single number by hand.
The authors introduce a new framework called Agentic Real2Sim, which acts like a team of digital detectives and engineers working together to solve this puzzle. Instead of a human manually cleaning up 3D models or guessing how heavy an object is, this system uses a "Vision-Language Agent" (an AI that can see images and understand language) to orchestrate the whole process. The system takes a video of a real robot interacting with objects—like a DROID robot picking up a tool—and breaks the job down into four specialized steps, each handled by a different "agent" in the team.
First, the Visual Processing Agent watches the video. It's like a forensic analyst who freezes the action to pick the best frames, cuts out the objects the robot is touching, and builds a 3D model of them. It figures out where the robot is, where the camera is, and what the floor looks like. If the AI gets confused by a shadow or a reflection, it can ask itself to try again with a different frame.
Next, the Physical-Prior Inference Agent steps in. This is the "physics teacher" of the group. It looks at the 3D models and guesses the invisible properties the computer needs to know to make the simulation feel real. It asks: "Is this object made of metal or rubber? How heavy is it? How does it bounce?" It attaches these physical rules to the digital models so that when the simulation runs, a heavy box falls differently than a light one.
Then, the Scene Preparation Agent acts as the stage manager. It takes all the 3D models, the robot's starting position, and the camera angles, and sets up the entire digital world in a physics engine called MuJoCo. It makes sure the robot's feet are on the ground and the objects are in the right spots.
Finally, the Simulator-in-the-Loop Agent is the quality control inspector. It runs the simulation to see if the robot can actually grab the object. If the robot misses the cup because the digital cup was placed a tiny bit too far away, this agent notices the failure, tweaks the position, and tries again until the robot succeeds. It's a loop of "try, check, fix, try again" that happens automatically.
The researchers tested this system on 100 different robot videos (from a dataset called DROID-100) involving tasks like picking up and placing objects. They found that the system could successfully convert about 48 out of 100 episodes into playable, realistic simulations where the robot could replay the task. While this means nearly half of the attempts still faced challenges or required further refinement, the results were promising. The system was also flexible enough to handle different types of challenges: it could simulate soft, squishy objects like cloth or ropes (deformable objects) and even the movements of a humanoid robot walking around.
One of the most exciting findings was about cost. The team tested the system using different "brains" (AI models) to run the agents. They found that a smaller, open-source AI model (a 31-billion-parameter model called Gemma 4) performed just as well as much more expensive, proprietary models from big tech companies. In fact, using the cheaper model cost about 31 times less money to run the same number of conversions. This suggests that we don't need the most expensive, powerful AI to build these digital twins; a smart, efficient agent can do the heavy lifting for a fraction of the price.
The paper concludes that while the system isn't perfect yet—it still struggles with some complex visual errors or tricky physics—it represents a major step forward. Instead of humans manually building simulations, we are moving toward a future where AI agents can watch a robot work, build a digital twin of that moment, and let other robots learn from it. The authors hope that in the future, these digital twins will be used to train robots to learn new skills faster and safer, turning the messy reality of the physical world into a playground for learning.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.