🤖 AI

Reward Learning through Ranking Mean Squared Error

This paper introduces Ranked Return Regression for RL (R4), a novel reward learning method that utilizes a ranking mean squared error loss to infer reward functions from human trajectory ratings, offering formal guarantees of minimality and completeness while empirically outperforming existing preference-based approaches on robotic benchmarks.

Original authors: Chaitanya Kharyal, Calarina Muslimani, Matthew E. Taylor

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chaitanya Kharyal, Calarina Muslimani, Matthew E. Taylor

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to walk, run, or pick up a cup. In the world of Artificial Intelligence (specifically Reinforcement Learning), the robot learns by trying things and getting points (rewards) for doing well. But here's the catch: designing the perfect point system is incredibly hard.

If you tell the robot, "Get points for moving forward," it might figure out a way to wiggle its legs on the ground to get points without actually walking. This is called "reward misspecification," and it's like giving a student a test where they can cheat by memorizing the answer key rather than learning the subject.

The Old Way: "Better or Worse?"

To fix this, researchers started asking humans for help. Instead of writing code, humans would watch the robot and say, "That walk was better than that one." This is called Preference-Based Learning.

Think of this like a blind taste test. You give a judge two cups of coffee and ask, "Which is better?" The judge picks one.

The Problem: This only gives you one tiny piece of information (a "yes" or "no"). It doesn't tell you how much better the coffee was. Also, if both cups are terrible, the judge can only say "they are equally bad," but they can't express that both are awful. It's a lot of work for the human to keep comparing things, and it doesn't capture the full picture.

The New Way: "Rate the Coffee"

A newer idea is Rating-Based Learning. Instead of comparing two things, you just show the human one thing and ask them to give it a score, like 1 to 5 stars.

The Benefit: This is easier for humans (less mental effort) and gives richer information. A 1-star rating tells you the coffee is terrible, while a 5-star tells you it's great. It captures the absolute quality, not just the relative one.

Enter R4: The "Ranking Score" Coach

The paper introduces a new method called R4 (Ranked Return Regression for RL). Think of R4 as a smart coach who uses a special scoring system to learn from those star ratings.

Here is how R4 works, using a simple analogy:

The Setup: Imagine you have a pile of robot walks. A human has rated them: "Bad," "Okay," and "Good."
The Old Method (RbRL): Previous methods tried to force the robot's internal score to match the exact middle of the "Good" bucket. It was like saying, "If you are rated 'Good,' your score must be exactly 75." This ignored the fact that some "Good" walks are barely good, while others are amazing. It forced all "Good" walks to look the same.
The R4 Method: R4 uses a clever trick called Ranking Mean Squared Error (rMSE).
- Instead of forcing scores to hit a specific number, R4 asks: "If I take one 'Bad' walk, one 'Okay' walk, and one 'Good' walk, can you rank them in the right order?"
- It uses a special mathematical tool (a "soft sorter") that can be tweaked by a computer. It looks at the robot's predicted scores for these three walks and asks, "Did you rank the 'Good' one higher than the 'Bad' one?"
- If the robot got the order wrong, the system gently nudges the robot's brain to fix the ranking.

Why is this better?

No Arbitrary Lines: You don't have to decide exactly where "Okay" ends and "Good" begins. You just say, "This is better than that."
Keeps Variety: It allows a "Good" walk to be a 90 or a 95. It doesn't force them all to be exactly 85. This preserves the natural differences between good performances.
Flexible: If a human wants to add a new category like "Super Good," the system can handle it without breaking.

The Proof: Theory and Reality

The authors didn't just guess this would work; they proved it mathematically.

The Guarantee: They showed that if the human's ratings make sense (i.e., a "Good" walk is truly better than a "Bad" one), R4 is guaranteed to find the best possible reward system that fits those ratings. It's the most efficient way to solve the puzzle without leaving out any valid solutions.

The Experiments: Robots and Humans

The team tested R4 in two ways:

Simulated Humans: They used computer programs to pretend to be humans rating robot walks. R4 consistently taught robots to move better and faster than the old methods.
Real Humans: They hired 8 real people to rate robot walks on a screen.
- The Result: Even though the humans were very different from each other (some used 4 stars, some used 12; some were strict, some were lenient), R4 still taught the robots to move better than the old methods.
- The Feeling: The humans reported that rating the robots was very easy and didn't feel like hard work (low "cognitive load").

The Bottom Line

The paper argues that R4 is a smarter, more flexible, and mathematically proven way to teach robots using human ratings. Instead of forcing humans to play "this is better than that," we let them give simple ratings, and R4 uses a special ranking trick to turn those ratings into a perfect teacher for the robot. It works better than previous methods, handles human quirks well, and is easy for people to use.

Technical Summary: Reward Learning through Ranking Mean Squared Error (R4)

Problem Statement
The application of Reinforcement Learning (RL) to real-world problems is frequently bottlenecked by the difficulty of reward design. Manual reward engineering is often an error-prone, trial-and-error process that can lead to reward misspecification, where agents learn unintended behaviors. While Reward Learning (inferring rewards from human feedback) offers an alternative, existing methods have limitations. Preference-based RL (PbRL), which relies on binary comparisons, is sample inefficient and provides only relative information without absolute quality assessment. Conversely, recent rating-based RL (RbRL) approaches allow humans to assign discrete, multi-class ratings (e.g., "bad," "neutral," "good"), offering richer feedback. However, existing RbRL methods, such as those using cross-entropy losses, impose rigid constraints on reward modeling, such as requiring explicit class boundaries and forcing trajectories within the same rating class to have identical predicted returns, thereby ignoring intra-class diversity.

Methodology: Ranked Return Regression for RL (R4)
The authors propose R4, a novel rating-based RL algorithm that learns reward functions from trajectory–rating pairs using a Ranking Mean Squared Error (rMSE) loss.

Core Mechanism: R4 treats human-provided discrete ratings as ordinal targets. During training, the algorithm samples one trajectory from each rating class. It computes the predicted discounted returns for these trajectories under the current reward model ( $\hat{r}_\theta$ ).
Differentiable Ranking: Instead of treating ratings as independent classification labels, R4 uses a differentiable sorting operator (specifically, soft ranks from Blondel et al., 2020) to rank the predicted returns. This produces a vector of soft ranks ( $\hat{R}$ ) corresponding to the sampled trajectories.
Loss Function: The objective is to minimize the Mean Squared Error between the soft ranks of the predicted returns and the human-provided rating classes ( $c$ ):
$L_{rMSE} = \frac{1}{n} \sum_{i=0}^{n-1} (\hat{R}(\tau_i) - c(\tau_i))^2$
This approach allows gradients to propagate through the ranking operation, enabling the reward model to adjust so that the relative ordering of predicted returns aligns with the ordinal human ratings.

Key Contributions

Algorithmic Innovation: The introduction of R4, which utilizes the novel rMSE loss to train reward functions from ordinal ratings without requiring explicitly defined class boundaries.
Theoretical Guarantees: The authors establish that under mild assumptions (deterministic human reward, hypothesis class coverage, and differentiable ranking approximating hard ranks), the solution set of the rMSE objective is both minimal and complete. This means the set of reward functions satisfying the rMSE loss is equivalent to the set of reward functions that perfectly satisfy the ordinal constraints imposed by the human ratings. In contrast, the authors prove that the human's true reward function is not guaranteed to be in the solution set of prior RbRL objectives.
Empirical Validation: R4 is validated in both offline and online settings using simulated ratings and a human-user study. It consistently matches or outperforms existing rating-based (RbRL) and preference-based (PEBBLE, SURF, QPA) methods across robotic benchmarks (OpenAI Gym and DeepMind Control Suite).
Human Study: An ethics-approved study with 8 participants demonstrated that R4 outperforms RbRL despite significant inter-participant variability in rating behavior. Participants reported low cognitive workload (mean NASA TLX score of 2.05/7), supporting the feasibility of rating-based feedback.

Results

Offline Setting: In OpenAI Gym domains (Reacher, Inverted Double Pendulum, Half Cheetah), reward functions trained with R4 led to statistically faster learning and higher final returns compared to RbRL when used to train a Soft Actor-Critic (SAC) agent.
Online Setting: In six DeepMind Control Suite environments, R4-trained agents learned significantly faster than all baselines in three environments and achieved higher final returns in four environments.
Human Feedback: In the user study, R4 achieved significantly faster learning and higher returns than RbRL in both Reacher and Hopper tasks, even when participants used varying numbers of rating classes and exhibited class imbalances.
Ablation: The study confirmed that R4 remains stable across different numbers of rating bins, whereas RbRL performance degrades as the number of bins deviates from an optimal range.

Significance
The paper claims that R4 represents a significant step forward in reward learning by providing a method that is both theoretically grounded and practically effective. By leveraging the ordinal nature of human ratings through a ranking-based loss, R4 avoids the restrictive assumptions of previous rating-based methods (such as fixed boundaries and intra-class homogeneity). The theoretical proof of minimality and completeness offers a rigorous justification for the algorithm's design, while the empirical results, including the human study, demonstrate its robustness to real-world feedback variability and its ability to reduce the cognitive burden on human raters compared to preference-based alternatives.