Scene2Demo: Self-Evolving Embodied Data Generation via Object-Action Graph
Scene2Demo is a self-evolving framework that leverages object-action graphs and feedback-driven refinement to automatically generate high-quality, executable embodied data from single images, significantly improving task planning success and enabling effective downstream robot policy learning.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Robots that can move through our homes and help with daily chores have long been a dream of science fiction, but building them has proven to be a stubbornly difficult reality. The core problem is not just teaching a machine how to move its arm, but teaching it how to understand a messy, unpredictable world and decide what to do next. To learn these skills, robots traditionally need vast amounts of data: thousands of hours of video showing a human performing a task, or millions of trial-and-error attempts in a computer simulation. Gathering this data by hand is slow, expensive, and often impossible for rare or dangerous scenarios. Researchers have turned to artificial intelligence to generate this data automatically, but early attempts often produced simulations that looked real but broke the laws of physics, or instructions that a robot could not actually follow. The challenge has been to create a system that can take a single photo of a room and a simple request, then build a working, physics-based simulation where a robot can successfully practice the task, failing and correcting itself until it gets it right.
A team of researchers has introduced a new system called SCENE2DEMO that tackles this problem by acting as a self-improving data factory. The process begins with a single photograph of a real-world room, such as a kitchen, and a simple text command from a user, like "pick up the glass." The system first uses advanced computer vision to reconstruct that photo into a fully interactive 3D simulation. It identifies the objects, their shapes, and where they are located, creating a digital twin of the scene that respects physical laws like gravity and collision. Once this virtual room is built, the system does not just guess how a robot should move; it breaks the user's request down into a logical sequence of small, manageable steps. It treats the task as a map, where each stop on the journey is a specific action, such as opening a door or lifting an object, and ensures that the end of one step perfectly sets up the beginning of the next.
The system then attempts to run this plan in the simulation. If the robot succeeds, the system records the entire sequence of movements and camera views as a perfect example for future learning. However, the true power of SCENE2DEMO lies in what happens when the robot fails. In many previous systems, a failure would simply be discarded. Here, the system employs a self-evolution mechanism. When a step goes wrong—perhaps the robot bumps into a door or drops an object—two specialized digital agents step in to investigate. One agent acts as a safety inspector, watching the video of the failure from multiple camera angles to pinpoint exactly what went wrong. The other agent acts as a supervisor, using that visual evidence to rewrite the plan. It might suggest moving the robot's base slightly to the left, or extending the arm a bit further, before trying the task again. This loop of trying, failing, analyzing, and correcting continues automatically until the robot completes the task successfully.
The researchers tested this approach on over one hundred different scenarios, ranging from simple tasks like picking up a cup to complex, multi-step chores like putting a glass into a refrigerator. Without the self-correction feature, the system succeeded in about 72 percent of the simple tasks. When they added the self-evolution loop for more complex, long-horizon tasks, the success rate improved significantly, and the quality of the individual steps became much more reliable. The system was able to generate high-quality training data that was then used to teach a real robot policy. When a robot learned from these automatically generated examples, it achieved a 96 percent success rate on opening a refrigerator and a 92 percent success rate on picking up a glass, proving that the data created by the machine was robust enough to teach a robot to perform the task in the real world.
This work suggests that we do not need to manually record every possible way a robot might interact with the world. Instead, by combining a realistic simulation with a feedback loop that allows the system to learn from its own mistakes, we can generate vast libraries of reliable training data. The system does not rely on magic or perfect vision; it relies on a structured process of breaking problems down, testing them, and refining the solution based on visual evidence. While the current system is limited to specific types of movements and objects, and it still struggles with the most complex physical constraints, it demonstrates a clear path forward. By automating the creation of training data, this approach could eventually allow robots to learn new skills quickly and safely, simply by looking at a picture of a room and being told what to do.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.