DREAM: Deployment-Time Demonstration Generation via Real-to-Sim for Scalable Policy Adaptation
DREAM is a framework that enables scalable adaptation of vision-language-action models to new workspaces by automatically generating fine-tuning data through real-to-sim reconstruction, LLM-driven task planning, and trajectory rendering, thereby eliminating the need for costly human teleoperation demonstrations.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Robots have long been masters of repetition, performing the same precise motion thousands of times in a factory, but they struggle when asked to adapt to a new room or a slightly different arrangement of objects. To teach a robot a new task, engineers traditionally rely on a method called teleoperation, where a human physically guides the machine through the steps, recording every movement to create a training dataset. While effective, this process is slow, expensive, and requires a person to be present for every single new environment. The field of robotics is now turning toward a different approach: using artificial intelligence models that can understand language and vision simultaneously. These models, trained on vast amounts of data, can already guess how to move a robot arm based on a simple sentence like "put the blue block in the bowl." However, even these advanced systems often fail when placed in a real-world setting they have never seen before, because the specific layout of the room, the lighting, and the exact position of the objects create a unique challenge that generic training cannot solve.
Researchers at the University of Tokyo have developed a system called DREAM to solve this problem without needing a human to guide the robot. Instead of asking a person to demonstrate the task, the system takes a short video of the empty workspace and a simple text instruction. It then builds a precise digital copy of that room, complete with the exact camera angles the robot will use to see. Using a sophisticated planning engine, the computer figures out a logical sequence of moves to achieve the goal, such as picking up an object and placing it somewhere else. It then generates thousands of variations of this task, simulating different starting positions for the objects and recording the robot's successful movements in this virtual world. These simulated movements are then turned into realistic images and action data, which are used to fine-tune the robot's brain before it ever touches the real world.
The core of this method lies in creating a "digital twin" of the physical space. The researchers start by recording a brief video of the table or workspace with a standard handheld camera. The system analyzes this footage to reconstruct the scene in three dimensions, capturing the texture and position of every surface and object. Crucially, it aligns this digital reconstruction with the robot's own coordinate system, ensuring that the virtual camera sees the world exactly as the real robot's cameras will. This allows the system to generate training data that looks and feels like the real environment, bridging the gap between the computer simulation and the physical world. Once the environment is built, the system uses a large language model to translate the human instruction into a set of logical goals. For example, if the instruction is to pack fruit, the system understands that it must first reach for the banana, then the peach, and finally place them on the plate.
With the goals defined, the system employs a task-and-motion planner to figure out the physical steps required to succeed. This planner acts like a highly logical engineer, breaking the task down into a sequence of actions and calculating the exact joint movements the robot needs to make to avoid collisions and reach its targets. It generates a small set of initial successful paths, known as source demonstrations. To create enough data to train a robust robot, the system then expands these few examples into a massive dataset. It takes the successful movements and applies them to hundreds of new, randomized scenarios where the objects are in different places. It checks each new attempt to ensure it is physically possible and safe, discarding any that fail. Finally, it renders these successful attempts into photorealistic video frames, creating a complete dataset of image-and-action pairs that the robot can learn from.
The researchers tested this approach on a real robot arm performing two distinct tasks: placing a blue block into a red bowl, and packing a banana and a peach onto a plate in a specific order. They compared robots trained on data generated by DREAM against robots trained on data collected by human operators guiding the machine. The results showed that the digital twin was a reliable predictor of real-world performance; if a robot performed well in the simulation, it performed well in the real world. More importantly, the system proved to be highly scalable. While a human operator could produce about one hundred demonstrations in a reasonable amount of time, the DREAM system generated one thousand high-quality training examples. When trained on this larger volume of automatically generated data, the robots achieved higher success rates than those trained on the smaller set of human demonstrations.
The study highlights a significant shift in how robots learn. While human teleoperation remains a valid way to collect data, it is limited by the time and attention of the operator. The DREAM system, by contrast, requires only a single video capture and a text instruction to produce a vast library of training examples. The initial setup takes a few minutes, but once the system is running, it can generate thousands of training scenarios in the time it takes a human to record just a handful. The researchers found that for simple tasks, the automatically generated data allowed the robot to succeed more often than the human-collected data, simply because the volume of practice was so much greater. For more complex, multi-step tasks, the system still outperformed human data collection, though the complexity of the task required more computational time to plan the initial movements.
This work suggests a future where robots can be deployed in new environments with minimal human intervention. Instead of waiting for a specialist to spend hours teaching a robot how to navigate a specific kitchen or workshop, a user could simply show the robot the room and tell it what to do. The system would then handle the heavy lifting of creating the training data, ensuring the robot is prepared for the specific challenges of that space. The researchers acknowledge that their current system works best with rigid objects on static tables, and future work will need to address more complex scenarios involving soft or moving objects. However, the ability to turn a simple video and a sentence into a fully trained robot policy represents a major step toward making robotic assistants truly adaptable and accessible for everyday use.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.