📊 statistics

Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning

This paper introduces Similarity as Reward Alignment (SARA), a robust contrastive framework for preference-based reinforcement learning that learns latent representations to compute rewards as similarities, demonstrating superior stability and noise resilience compared to baselines on continuous control benchmarks.

Original authors: Sara Rajaram, R. James Cotton, Fabian H. Sinz

Published 2026-07-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sara Rajaram, R. James Cotton, Fabian H. Sinz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to walk, dance, or cook a meal. In the world of artificial intelligence, this is called Reinforcement Learning. Usually, the robot learns by trying things and getting a score, or "reward," from a human programmer. If the robot does something good, it gets a point; if it falls over, it loses one. But here's the catch: writing the perfect scoring system is incredibly hard. It's like trying to write a rulebook for "good cooking" that covers every possible mistake a chef could make without accidentally telling them to burn the house down.

To solve this, scientists developed a clever shortcut called Preference-based Reinforcement Learning (PbRL). Instead of writing a rulebook, they just ask humans to look at two different robot attempts and say, "I like this one better than that one." The robot then tries to figure out the hidden rules of what makes the "good" one better. It's a lot easier to say "I prefer this pizza" than to write a manual on exactly how much cheese is perfect. However, there is a big problem: humans aren't perfect. We get tired, we get confused, and sometimes we make mistakes. If the robot learns from a teacher who is often wrong, the robot might learn to be clumsy or dangerous. This paper tackles the messy reality of teaching robots with human teachers who sometimes get it wrong.

The researchers behind this study, Sara Rajaram, R. James Cotton, and Fabian H. Sinz, noticed that most current methods for teaching robots from human preferences rely on a specific mathematical trick called the Bradley-Terry model. Think of this model like a strict referee who only cares about head-to-head matches. If you tell the referee, "Robot A is better than Robot B," the referee assumes this is an absolute truth and adjusts the robot's brain accordingly. But what if you were just guessing? Or what if you were tired and picked the wrong one? The old model treats every mistake as a fact, which can confuse the robot and make it perform terribly when the human teacher slips up.

To fix this, the team introduced a new method called SARA (Similarity as Reward Alignment). Instead of acting like a strict referee who only looks at pairs, SARA acts more like a wise art critic who looks at a whole gallery of paintings. SARA doesn't just compare two robots at a time; it looks at all the "good" examples the human liked and finds the common patterns that make them special. It creates a mental "fingerprint" of what a perfect performance looks like. Then, when the robot tries something new, SARA asks, "How much does this look like our fingerprint?" If it looks similar, the robot gets a reward. If it looks different, it gets a lower score.

The beauty of SARA is that it ignores the noise. If a human accidentally says a bad performance is "good," SARA doesn't panic. Because it has looked at so many other "good" examples, it realizes that this one bad example doesn't fit the pattern of the rest. It's like a music teacher who has heard thousands of perfect piano pieces; if a student plays a song with a wrong note but the teacher accidentally claps, the teacher's brain knows, "Wait, that didn't sound like the other great songs," and doesn't let the student think they nailed it.

The team tested this idea on a variety of robot tasks, like making a virtual hopper jump or a walker move forward. They deliberately messed up the human feedback, telling the robots that bad moves were actually good, with error rates ranging from 0% up to 40%. In these tests, the old methods (the strict referees) started to fall apart as the errors increased, with their performance dropping by as much as 73% on some tasks. SARA, however, stayed steady. Even when the human feedback was noisy, SARA's robots kept performing well, showing statistically significant improvements over the older methods.

The researchers also checked how well SARA understood the real goal of the task, not just what the human said. They found that the rewards SARA calculated were much more closely aligned with the actual physics of the environment than the rewards calculated by the older methods. This suggests that SARA is better at filtering out the human's confusion and finding the true "good" behavior hidden underneath.

In short, this paper suggests that when teaching robots with human feedback, it's better to look for the "vibe" of a good performance across many examples rather than obsessing over every single comparison. By using a method that learns from the collective wisdom of a group of "good" examples rather than just pairwise fights, SARA makes robots much more robust against human mistakes. It doesn't mean the human teacher has to be perfect; it just means the robot is smart enough to know when the teacher is having a bad day.

Technical Summary: Similarity as Reward Alignment (SARA)

Problem Statement

Preference-based Reinforcement Learning (PbRL) aims to align agent behavior with human intent by learning from human feedback on trajectory pairs, thereby alleviating the burden of manual reward engineering. However, existing PbRL methods predominantly rely on the Bradley-Terry (BT) model to learn reward functions from pairwise rankings. This approach faces significant limitations in real-world scenarios where human labelers are non-experts or operate under time constraints, leading to inevitable labeling errors.

Prior research indicates that BT-based models are fragile to noise; they assume transitive preferences (an assumption often violated in human psychology) and can overfit to relative rankings in trajectory pairs. Empirical studies suggest that even 10% label error rates can significantly degrade RL performance, with realistic disagreement rates among human labelers often reaching 25%. Furthermore, formal robustness proofs for BT models under noisy settings remain an open challenge. The core problem addressed is the lack of robustness in current PbRL frameworks when faced with noisy, non-expert preference data.

Methodology: Similarity as Reward Alignment (SARA)

The authors propose SARA, a contrastive learning framework that shifts the paradigm from learning pairwise relative rankings to learning a latent representation of the entire set of preferred behaviors.

Core Architecture

SARA employs a two-stage Transformer-based encoder:

Trajectory Encoding (Enc 1): Each trajectory (preferred $\sigma_p$ or non-preferred $\sigma_n$ ) is processed through a Transformer encoder with positional encoding. Average pooling over timesteps produces a single latent representation ( $u_{p,i}$ or $u_{n,i}$ ) for each trajectory.
Set Encoding (Enc 2): The trajectory representations are randomly partitioned into $k$ subsets (where $k=2$ in experiments) within each category (preferred and non-preferred). A second Transformer encoder processes these subsets, allowing attention within the subset to aggregate information. This produces set-level latent representations ( $z^k_p$ and $z^k_n$ ).

Training Objective

The model is trained using the SimCLR contrastive loss. The objective is to:

Pull together the latent representations of different subsets of the same category (e.g., $z^1_p$ and $z^2_p$ ).
Push apart the representations of preferred subsets from non-preferred subsets ( $z^k_p$ vs $z^k_n$ ).

This randomization of subset composition at every epoch forces the encoder to learn generalizable patterns distinguishing preferred from non-preferred behaviors, rather than overfitting to specific trajectory pairings or noisy labels.

Reward Inference

Unlike BT-based methods that output a reward function parameterized by $\psi$ , SARA learns a frozen preferred set representation ( $z^*_p$ ) by encoding the entire dataset of preferred trajectories. During downstream RL training, the reward for a sampled trajectory $\sigma_t$ at timestep $t$ is computed as the cosine similarity between the trajectory's latent representation ( $z_t$ ) and the frozen preferred latent:
$r_t = \cos(z_t, z^*_p)$
Theoretical justification draws on recent work (Sun et al., 2025) suggesting that predicting the probability of a single trajectory being preferred is sufficient for alignment, avoiding the pitfalls of relative ranking models.

Key Contributions

Robust Contrastive Framework: SARA introduces a novel method that encodes preferred and non-preferred trajectory sets into latent representations, computing rewards via similarity to the preferred set. This avoids the pairwise ranking assumptions of the BT model.
Noise Resilience: By learning an attention-weighted representation of the entire preferred set, SARA achieves robustness to labeler errors. It does not rely on fitting pairwise rankings, which are susceptible to corruption by noisy labels.
Mechanistic Analysis: The authors provide a mechanistic explanation for SARA's robustness, supported by t-SNE visualizations and ablation studies. They demonstrate that the encoder learns to downweight mislabeled trajectories that do not share patterns with the majority of the preferred set, effectively filtering noise without explicit filtering mechanisms.
Statistical Validation: The paper provides rigorous statistical evidence (Wilcoxon signed-rank test, $p < 0.01$ ) demonstrating SARA's superiority over baselines across varying noise levels.

Experimental Results

The authors evaluated SARA on continuous control offline RL benchmarks (D4RL: Mujoco locomotion, Franka Kitchen, Adroit) using preference datasets with injected noise rates of 0%, 10%, 20%, and 40%.

Performance Stability: SARA demonstrated consistent policy evaluation returns across 0–20% error rates. In contrast, baselines (Preference Transformer, PT+ADT, DPPO) showed significant degradation, with some dropping by up to 73% on specific datasets as error rates increased.
Statistical Significance: A Wilcoxon signed-rank test across 24 task-condition cells confirmed that SARA's performance advantage over all baselines is statistically significant ( $p < 0.01$ ).
Reward Correlation: SARA-inferred transition rewards showed higher Pearson correlation with the true environment rewards (used as a proxy for underlying preference criteria) compared to baselines across all noise rates. This indicates SARA better captures the true preference criteria even when labels are noisy.
Ablation Studies: Removing the set-based encoding (training on individual trajectory pairs via contrastive learning) resulted in significantly lower and less stable returns, confirming that the set-level aggregation is critical for robustness.

Significance and Claims

The paper claims that SARA offers a more robust alternative to standard BT-based PbRL methods, particularly in scenarios involving non-expert labelers or time-constrained annotation where error rates are high.

Shift in Paradigm: The authors argue that SARA's focus on representation learning and set-level consensus, rather than pairwise relative rankings, addresses the fundamental fragility of BT models to noise and intransitive human preferences.
Practical Utility: While SARA may not displace BT models in high-quality, expert-heavy data scenarios (where BT can still perform well), it provides a stable baseline when annotation quality is uncertain.
Versatility: The framework is adaptable to diverse feedback formats and does not strictly require paired data, allowing for the use of subsets of preferred samples from a larger pool.

The authors remain modest regarding their claims, acknowledging that SARA does not guarantee transitive preference orderings (a known limitation of human preferences) and that its exclusive use of D4RL environments limits generalizability to other benchmarks. They position SARA not as a universal replacement for BT, but as a necessary evolution for robustness in realistic, noisy preference learning settings.