Residual Reward Models: Leveraging Prior Knowledge for Efficient Preference-based Reinforcement Learning in Robotics

Teaching a robot to perform a complex physical task, like pressing a button or picking up a delicate object, is a difficult puzzle for engineers. In the world of robotics, the machine needs a set of instructions that tells it what it is doing well and what it is doing poorly. This set of instructions is called a reward function. If the instructions are vague or wrong, the robot might learn to bypass the system, finding a way to get a high score without actually completing the task, or it might simply give up because it cannot figure out what is expected. Traditionally, humans have to write these instructions by hand, guessing which movements will lead to success. This process is slow, expensive, and often fails when the robot encounters a situation the human did not anticipate.

A newer approach, known as preference-based learning, tries to solve this by letting humans simply say which of two robot movements looks better, rather than writing complex rules. While this sounds simpler, it creates a new problem: the robot needs to see thousands of these comparisons to learn anything useful. In a real-world setting, asking a human to watch and judge a robot that many times is impractical and costly. The robot learns too slowly, and the cost of human attention becomes a bottleneck that stops these machines from being useful outside of a controlled lab.

Researchers at the University of Toronto and Tsinghua University have developed a method to speed this process up significantly. They call their approach the Residual Reward Model. Instead of starting from scratch and asking a human to teach the robot everything from zero, this method allows the robot to start with a "best guess" about how the task should be done. This guess can come from a simple rule written by an engineer, a description generated by a large language model, or even a reward function learned from watching a human perform the task once. The robot then uses this initial guess as a foundation. When a human provides a preference, saying "this movement was better than that one," the robot does not try to rewrite the entire rulebook. Instead, it learns only the small difference, or the "residual," needed to correct the initial guess.

Think of it like a student who already knows the basic rules of a sport but needs a coach to point out the specific nuances of their technique. The student does not need to relearn how to run or throw; they only need to adjust their form slightly to match the coach's feedback. In the same way, the robot uses the human's feedback to make small, precise adjustments to its pre-existing understanding of the task. This structure keeps the learning process stable. If the initial guess is imperfect, the robot can still learn because it only has to correct the errors rather than discover the entire concept from nothing.

The researchers tested this idea in a variety of simulated environments, including tasks where a robotic arm had to press buttons, sweep objects into a bin, or unlock doors. They also tested it on a real physical robot, a Franka Panda arm, in a laboratory setting. In every case, the method that used the "best guess" plus the small corrections learned much faster than methods that tried to learn everything from human preferences alone. In the simulations, the robot using this new method reached a perfect success rate in many tasks while requiring far fewer human judgments. For example, in a task where a robot had to press a button, a standard method that tried to learn from scratch stalled at a twenty percent success rate, while the new method reached one hundred percent.

The study also showed that this approach is robust against mistakes. If the initial "best guess" was slightly wrong, or if the human feedback was occasionally noisy or inconsistent, the robot could still learn the correct behavior. The initial guess acted as a safety net, preventing the robot from wandering into useless behaviors, while the corrections ensured it eventually found the right path. This was particularly important when the researchers tested the system with very limited human feedback. Even when the human was asked to provide only a tiny number of judgments, the robot using the residual method continued to improve, whereas the standard method failed to learn anything useful.

Perhaps most importantly, the researchers demonstrated that this learning could transfer directly from a computer simulation to a real physical robot without any additional tuning. They trained the robot in a virtual environment and then placed the learned policy onto a real Franka Panda arm to perform tasks like reaching for an object, pushing a block, or picking something up and moving it. The robot trained with the residual method succeeded in these real-world tasks much faster than the baseline method. In one difficult task involving pushing an object, the standard method managed to succeed only half the time after extensive training, while the new method achieved a ninety-five percent success rate with the same amount of training time.

The findings suggest that by combining prior knowledge with human feedback, robots can learn complex physical skills with far less human supervision than previously thought possible. This does not mean the robot knows everything beforehand, but rather that it uses what it does know as a starting point to make the most of every piece of human feedback it receives. This efficiency could be a key step toward making robotic assistants practical for real-world jobs, where time and human attention are limited resources. The work confirms that giving a robot a head start, even a rough one, allows it to learn from human preferences in a way that is both faster and more reliable than starting from a blank slate.