🤖 AI

Reward Model Interpretability via Optimal and Pessimal Tokens

Original authors: Brian Christian, Hannah Rose Kirk, Jessica A. F. Thompson, Christopher Summerfield, Tsvetomira Dumbalska

Published 2026-02-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Brian Christian, Hannah Rose Kirk, Jessica A. F. Thompson, Christopher Summerfield, Tsvetomira Dumbalska

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a robot that is supposed to be helpful, harmless, and honest. To teach it how to be "good," you don't just program rules; you hire a team of human judges. These judges look at two different answers the robot might give and say, "I like this one better."

The computer then builds a special "Scorekeeper" (called a Reward Model) to learn from these judges. This Scorekeeper's job is to look at any sentence and give it a single number: a high score for "good" and a low score for "bad." Once this Scorekeeper is trained, it is used to teach the robot how to talk to millions of people.

This paper is like a detective story where the authors decide to stop looking at the robot and start interrogating the Scorekeeper itself. They wanted to see: Does this Scorekeeper actually understand human values, or is it just memorizing weird tricks?

Here is what they found, explained through simple analogies:

1. The "Taste Test" Experiment

The researchers decided to test the Scorekeepers by asking them a simple question: "What is the greatest thing ever?"

Instead of just asking for a few answers, they asked the Scorekeepers to rate every single word in their entire dictionary (about 100,000 to 250,000 words). It's like asking a food critic to taste every single ingredient in a supermarket and rank them from "Best" to "Worst."

The Shocking Discovery:
Even though all these Scorekeepers were trained to do the same job, they had completely different tastes.

One model thought "Love" was the best thing.
Another thought "Freedom" was the best.
A third thought "Sonder" (a fancy word for realizing strangers have complex lives) was the top prize.

It's as if you hired ten different food critics to judge a burger, and one gave it a 10/10, another gave it a 2/10, and a third said, "Actually, I prefer a rock." This proves that these "Scorekeepers" are not interchangeable. You can't just swap one for another and expect the robot to behave the same way.

2. The "Framing" Trick (The Mirror Effect)

The researchers noticed something strange about how the Scorekeepers react to the tone of the question. This is similar to a psychological trick humans fall for.

Scenario A: You ask, "Which vacation spot is the best?" The Scorekeepers get very excited about positive words (like "sunshine" or "beach") and ignore the negative ones.
Scenario B: You ask, "Which vacation spot is the worst?" Suddenly, the Scorekeepers flip their script. They become hyper-sensitive to negative words (like "rain" or "traffic") and ignore the positive ones.

The Metaphor: Imagine a security guard at a club. If you ask, "Who looks cool?" the guard checks for sunglasses and smiles. If you ask, "Who looks dangerous?" the guard suddenly ignores the smiles and only checks for knives. The guard isn't looking at the person; they are reacting to the question. The paper found that these AI Scorekeepers do the same thing: they don't have a stable sense of "good" or "bad"; they just react to how the question is framed.

3. The "Familiarity Bias" (The Mere-Exposure Effect)

The study found that the Scorekeepers liked words simply because they were common.

If a word appears often in books and on the internet, the Scorekeeper gives it a higher score, even if the word isn't particularly "good."
If a word is rare, it gets a lower score.

The Analogy: It's like a music critic who thinks a song is a masterpiece just because they've heard it on the radio 1,000 times, while ignoring a brilliant new song they've never heard. The paper suggests the Scorekeepers are leaking this "popularity bias" from their training data, rather than judging the actual value of the words.

4. The "Silencing" of Identity Groups

This is the most concerning finding. The researchers found that when they asked about "the greatest thing ever," the Scorekeepers gave very low scores to words related to specific groups of people (like "Black people," "Jews," or "homosexuals").

The Metaphor: Imagine a teacher grading essays. If a student writes about a specific group of people, the teacher automatically gives them a failing grade, even if the essay is well-written and positive. The paper suggests this happens because the Scorekeepers were trained to be "harmless." In their training, words about these groups often appeared in bad contexts (hate speech, news about violence, etc.). So, the Scorekeeper learned to treat the words themselves as dangerous, effectively "erasing" them from the list of "good" things.

5. The "Human vs. Machine" Mismatch

Finally, the researchers compared the Scorekeepers' rankings to a massive database of real human opinions (called EloEveRything), where thousands of real people voted on what they liked.

Humans ranked "The Universe," "Water," and "Knowledge" as the greatest things.
The Scorekeepers often ranked these low. Instead, they loved abstract feelings like "Unconditional love" or "Imagination."
The Big Gap: When it came to sensitive topics like "Sex" or "Black people," the Scorekeepers ranked them much lower than real humans did.

The Takeaway: The Scorekeepers are not perfect mirrors of human values. They are distorted mirrors. They seem to be "over-correcting" to avoid being offensive, which leads them to accidentally punish innocent or neutral words related to identity and sexuality.

Summary

The paper concludes that these "Reward Models" are not the neutral, perfect judges we thought they were. They are:

Inconsistent: Different models have different, conflicting ideas of what is "good."
Framing-Sensitive: They change their minds based on how you ask the question.
Biased: They favor common words and unfairly punish words related to certain identity groups.

The authors warn that if we use these flawed Scorekeepers to train the AI robots that millions of people talk to every day, we might accidentally bake these weird biases and distortions into the robots' personalities.

Technical Summary: Reward Model Interpretability via Optimal and Pessimal Tokens

Problem Statement
Reward models (RMs) are central to the alignment of Large Language Models (LLMs) with human values, serving as the primary mechanism for converting prompt-response pairs into scalar rewards during Reinforcement Learning from Human Feedback (RLHF). Despite their critical role as the "human value rubber meeting the road," reward models themselves remain under-explored. They are often treated as disposable intermediaries, yet they represent the most direct encoding of human value judgments in AI systems. Current literature lacks a comprehensive understanding of the consistency, faithfulness, and internal properties of these models, particularly regarding how they encode complex, context-dependent values and whether different models trained on similar objectives are interchangeable.

Methodology
The authors propose a novel approach to reward model interpretability through an exhaustive analysis of model responses across their entire vocabulary space. The study focuses on ten recent open-source reward models of varying parameter counts (2B to 27B) and architectures, selected from the RewaRdBench leaderboard.

The core methodology involves:

Exhaustive Single-Token Search: The researchers constructed a prompt ("What, in one word, is the greatest thing ever?") and computed the scalar reward score for every possible single token in each model's vocabulary (ranging from ~128,000 to ~256,000 tokens). This allowed for a complete ranking of "optimal" (highest-scoring) and "pessimal" (lowest-scoring) tokens.
Quantitative Analysis: The study analyzed the statistical properties of these reward distributions, including skewness, variance, and ordinal correlations (Kendall's $\tau$ ) between models. They employed Multidimensional Scaling (MDS) and Representational Similarity Analysis (RSA) to determine how factors like base model, developer, parameter count, and leaderboard ranking influence model similarity.
Framing and Sentiment Analysis: The authors tested the models with positively framed prompts ("best thing ever") and negatively framed prompts ("worst thing ever") to investigate sensitivity to sentiment valence and framing effects. They also controlled for word frequency to test for a "mere-exposure effect."
External Validation: To assess faithfulness to human values, the authors compared model rankings against an independent ground-truth dataset, EloEveRything, which aggregates pairwise human preferences over Wikipedia-derived entities.
Multi-Token Optimization: Using Greedy Coordinate Gradient (GCG) optimization, the authors extended the search to multi-token sequences (2–5 tokens) to identify optimal and pessimal responses beyond single tokens.

Key Contributions

Exhaustive Vocabulary Analysis: The paper pioneers a complete ranking of reward scores for all tokens in the vocabulary of ten diverse reward models, revealing granular characteristics of their value encoding.
Systematic Asymmetries: The authors demonstrate that reward distributions exhibit positive skew and systematic asymmetries in sensitivity. Models are more sensitive to distinctions among high-scoring (positive sentiment) tokens than low-scoring ones.
Framing Effects: The study establishes that reward models mirror human cognitive biases regarding framing. Sensitivity to token sentiment shifts based on prompt framing; models are more sensitive to positive sentiment in positive frames and negative sentiment in negative frames.
Frequency Bias: The research identifies a "mere-exposure effect," where more frequent words in the English language receive higher reward scores, suggesting a leakage from pre-trained base models into the reward signal.
Heterogeneity and Bias: The work reveals significant heterogeneity between models trained on similar objectives, invalidating the assumption of their fungibility. It also uncovers systematic devaluation of identity group references (e.g., "Black people," "Jews," "homosexuals") and sexual content, even in response to positive prompts.

Results

Model Heterogeneity: There is substantial diversity in token rankings across models, even those from the same developer or with similar architectures. Kendall's $\tau$ correlations between models range from 0.22 to 0.39 when compared to human rankings, and 0.55 between models themselves.
Base Model Influence: RSA indicates that the choice of base model and the model's ranking on RewaRdBench are the primary drivers of similarity in token rankings, suggesting that RMs inherit idiosyncratic biases from their pre-trained foundations.
Framing Sensitivity: When prompted with "the worst thing ever," models became more sensitive to negative-sentiment tokens, reversing the bias observed in positive prompts. This suggests RMs do not treat "best" and "worst" as simple inverses but operate in a higher-dimensional space.
Identity Group Devaluation: In the EloEveRything comparison, reward models systematically under-valued concepts related to nature, technology, and specific identity groups compared to human raters. For instance, "Black people" was ranked significantly lower by models than by humans, and terms like "rape" and "Hitler" appeared as pessimal tokens, while identity slurs were also heavily penalized.
Multi-Token Findings: GCG optimization confirmed that programming artifacts and malformed code receive low scores, while subjective, non-committal answers often score highly. Notably, identity group terms like "Jews" emerged as pessimal tokens for both "greatest" and "worst" prompts, reinforcing the linguistic erasure effect.

Significance and Claims
The paper argues that reward models are not merely neutral proxies for human preference but are complex artifacts that encode specific, and sometimes concerning, biases. The authors claim that:

Interchangeability is a Myth: Reward models trained on similar objectives are not interchangeable; they exhibit significant heterogeneity that can lead to different downstream behaviors in fine-tuned LLMs.
Unintended Consequences of Harmlessness: The systematic devaluation of identity groups and sexual content appears to be an unintended consequence of "harmlessness" training objectives, potentially leading to linguistic erasure rather than safe engagement.
Framing Risks: The sensitivity to framing suggests that current RLHF practices (which typically ask raters to choose the "better" response) may inadvertently bias the reward model's dynamic range, potentially undermining the goal of steering models away from harmful outputs in negative contexts.
Need for Robust Proxies: The findings challenge the suitability of current reward models as proxies for complex human values and highlight the need for more robust, transparent, and carefully curated value encoding to prevent the propagation of biases into widely deployed LLMs.

The authors conclude that while reward models are essential tools for alignment, they require rigorous interpretability analysis to ensure they faithfully represent human values without introducing new, systematic distortions.