WARP-RM: A Warp-Augmented Relative Progress Reward Model for Data Curation
The paper introduces WARP-RM, a fully self-supervised reward model that generates dense relative progress signals from time-warped successful demonstrations to enable WARP-BC, a behavior cloning method that significantly improves robot performance on long-horizon tasks by filtering and reweighting action chunks from mixed-quality data.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Robots have long struggled to learn from watching humans. While a person can glance at a task and understand the flow of movement, a robot trained on video data often learns the mistakes along with the successes. If a human operator pauses to think, fumbles with an object, or tries a clumsy grip before succeeding, the robot sees these hesitations as part of the correct path. This is a particular problem for long, complex tasks like folding laundry or assembling objects, where a single moment of confusion can derail the entire sequence. For years, researchers have tried to teach robots to ignore these bad moments, but most methods required expensive human labels to mark exactly where a mistake happened, or they discarded entire video clips if they contained any errors, throwing away valuable recovery moves hidden inside imperfect demonstrations.
A team of researchers at the University of California, Berkeley, and Stanford University has developed a new way to teach robots to distinguish between productive movement and wasted time without needing any human labels. They created a system called WARP, which stands for Warp-Augmented Relative Progress. Instead of asking a human to watch hours of video and draw boxes around good and bad moments, the researchers taught the computer to understand the speed and direction of progress by playing the videos back at different speeds and even in reverse. By training the robot to recognize what "forward" progress looks like compared to "backward" or stalled movement, the system learned to assign a score to every single frame of a video. This score tells the robot how fast the task is moving forward, if it is stuck, or if it is actually moving backward.
The researchers tested this idea on a physical robot with two arms, asking it to fold T-shirts that were crumpled in a bin. They created training datasets of varying quality. In the best datasets, the human demonstrations were fast and efficient. In the worst datasets, the human demonstrations were full of pauses, retries, and long periods of fumbling. When they trained a standard robot learning system on these messy, low-quality videos, the robot failed almost completely. It would get stuck in loops of tiny, useless adjustments, unable to finish the task. However, when they used the new WARP system to filter the data, the results changed dramatically. The system automatically identified the slow, hesitant parts of the videos and reduced their importance, while keeping the fast, decisive moments. On the messy datasets where the standard robot failed twenty times out of twenty, the WARP-trained robot succeeded nineteen times out of twenty. It also folded shirts nearly eighteen times faster than the standard robot when given the same poor-quality data.
The team did not stop with T-shirts. They also tested the method on a task where the robot had to place plastic bottles into a bin. In the real world, the new system placed seventy-four out of eighty bottles, while the standard system managed only fifty-nine. The new robot placed the bottles faster and with more consistency. To be sure these results were not just a fluke of the specific robot hardware, the researchers ran a massive simulation with five hundred twelve different scenarios. In this virtual test, the new system placed 290 bottles per hour, significantly outperforming the standard system, which managed 237 bottles per hour. Even when compared to other advanced methods that try to clean up data, the new approach consistently produced the fastest and most reliable results.
A key part of why this works is how the system learns to recognize progress. The researchers did not tell the computer what a "folded shirt" looks like. Instead, they took successful videos of humans folding shirts and played them back at random speeds, sometimes slowing them down to a crawl and sometimes speeding them up. They also played some videos in reverse. The computer was then asked to guess how much time had passed between the start of a clip and the current moment. By learning to predict the time elapsed in these warped, scrambled versions of the video, the computer learned to understand the natural rhythm of the task. It learned that a fast, smooth motion usually means the task is moving forward, while a slow, jerky motion or a backward movement means the task is stalled or failing. This allowed the system to generate a continuous score for every frame of video, telling the robot exactly which parts of the demonstration to pay attention to and which to ignore.
The researchers found that simply throwing away bad videos was not enough. The most effective strategy was to keep the messy videos but change how the robot learned from them. The system would take a chunk of action from the video and look at the progress score at the very end of that chunk. If the score was high, meaning the task was moving forward quickly, the robot would learn from that chunk with full intensity. If the score was low or negative, meaning the robot was hesitating or moving backward, the system would either ignore that chunk entirely or teach from it very lightly. This approach allowed the robot to learn from the recovery moves humans made when they dropped a shirt and picked it up again, without learning the panic and hesitation that caused the drop in the first place.
This work suggests that robots do not need perfect human demonstrations to learn complex skills. They can learn from messy, real-world data if they have a way to understand the flow of time and progress within that data. The researchers noted that their method relies on a specific type of data augmentation where videos are played in reverse, which is physically impossible for a real robot to do. However, the system still learned useful patterns from this artificial training. While the method is not a magic solution that works for every possible task, it provides a powerful new tool for teaching robots to ignore the noise of human error and focus on the signal of successful action. The team has made their code and data available so other researchers can test these ideas on their own robots and tasks, potentially leading to a new generation of robots that can learn quickly from the imperfect, everyday world.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.