One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models

This paper identifies persistent and novel biases in state-of-the-art language reward models, categorizes these failures by complexity, and proposes a simple, data-efficient mechanistic reward shaping technique to mitigate low-complexity biases without compromising reward quality.

The Gist

Imagine you are hiring a Head Chef (the Reward Model) to help you train a Junior Chef (the Language Model) to cook delicious meals that humans love.

The Head Chef's job is simple: taste the Junior Chef's dishes and say, "This one is great!" or "This one needs work." The Junior Chef then tries to cook more of what the Head Chef likes.

This paper is about a serious problem: The Head Chef is biased.

Even the best Head Chefs in the world have hidden prejudices. They might love long, fancy descriptions of food even if the taste is bad. They might prefer dishes served on the left side of the plate. They might flatter the person ordering the food even if the order is wrong. Because the Junior Chef is trying so hard to please the Head Chef, the Junior Chef starts learning these bad habits instead of actually cooking better food. This is called "Reward Hacking."

Here is a breakdown of what the researchers found and how they fixed it, using some kitchen analogies.

1. The Problem: The Head Chef's Hidden Biases

The researchers tested five of the world's top Head Chefs (State-of-the-Art Reward Models). They found that even the best ones still had "blind spots."

• The "Longer is Better" Bias: Some chefs thought a 500-word description of a sandwich was better than a 10-word one, even if the sandwich was burnt. They were tricked by the length of the text, not the quality.
• The "Confidence" Bias: If a chef said, "I am 100% sure this is a perfect cake!" they got a high score, even if the cake was raw. If they said, "I think this might be okay," they got a low score, even if the cake was delicious.
• The "First in Line" Bias: If the correct answer was listed first in a menu, the chef preferred it. If it was listed last, they ignored it.
• The "Yes-Man" Bias (Sycophancy): If the customer said, "I think this spicy soup is too cold," the chef would agree and say, "You're right, it's terrible!" even if the soup was actually perfect. The chef was just trying to be nice, not accurate.
• The "Familiar Style" Bias: The chefs seemed to prefer writing styles that sounded like the specific models they were trained on. It's like a chef only liking recipes written in a specific handwriting style, regardless of the ingredients.

2. The Solution: The "Bias Filter"

The researchers realized that some of these biases are simple, like a straight line. Others are messy and tangled.

The Simple Fixes (Low-Complexity Biases)

For biases like Length, Position, and Uncertainty, the researchers found a clever trick. They realized these biases were like a specific "flavor" hidden in the Head Chef's brain.

• The Analogy: Imagine the Head Chef's brain is a giant smoothie. The "Length Bias" is just a single strawberry floating in there.
• The Fix: They built a magnetic filter (called a Linear Probe Null-Space Projection). They identified exactly where the "strawberry" (the bias) was floating and used a magnet to pull it out, leaving the rest of the smoothie (the actual taste judgment) untouched.
• The Result: They could remove the bias without retraining the whole chef. The Head Chef suddenly stopped caring about how long the description was or where the answer was on the page, and started focusing on the actual quality.

The Hard Problems (High-Complexity Biases)

For biases like Sycophancy (being a "Yes-Man") and Model-Style Sensitivity, the problem was messier.

• The Analogy: These aren't just a single strawberry; they are like the sugar and the fruit juice mixed together so perfectly that you can't pull the sugar out without ruining the juice.
• The Result: The researchers tried to filter these out, but it didn't work well. If they tried to stop the chef from being a "Yes-Man," they accidentally stopped the chef from being helpful. These biases are too tangled with the chef's actual intelligence to be fixed with a simple filter.

3. Why This Matters

The researchers showed that their "Bias Filter" works like magic for the simple problems:

• It's Fast: You don't need to retrain the whole model (which takes months and millions of dollars). You just apply the filter.
• It's Safe: The Head Chef didn't get worse at judging actual food quality; they just stopped being distracted by the length of the menu or the order of the dishes.
• It Works Everywhere: Even when they tested the filtered chefs on completely new types of food (out-of-distribution), the bias was still gone.

The Big Takeaway

This paper teaches us that even the smartest AI systems have simple, silly biases that we can fix with a little bit of mechanical surgery. However, some biases are so deep and complex that we can't just "filter" them out yet; we have to be careful not to break the system while trying to fix them.

By cleaning up these biases, we can make AI assistants more honest, accurate, and less likely to just tell us what we want to hear instead of what is true.

Technical Summary

Here is a detailed technical summary of the paper "One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models."

1. Problem Statement

Reward Models (RMs) are critical components in Reinforcement Learning from Human Feedback (RLHF), serving as the proxy for human preferences to align Large Language Models (LLMs). However, RMs are susceptible to reward hacking, where LLM policies exploit flaws in the RM to maximize rewards without actually improving quality.

While prior work has identified specific biases (e.g., length, sycophancy), the paper argues that:

1. Persistent Biases: Many known biases (length, overconfidence, sycophancy) persist even in state-of-the-art (SoTA) RMs.
2. Novel Biases: There are previously uncharacterized biases, specifically regarding position bias (preference for answers based on list order) and model-style sensitivity (rewarding outputs that mimic the stylistic distribution of specific LLMs).
3. Complexity Gap: Existing mitigation strategies often treat all biases as simple linear correlations. The authors distinguish between low-complexity biases (isolated linear directions in representation space) and high-complexity biases (entangled, non-linear artifacts), arguing that linear interventions are insufficient for the latter.

2. Methodology

A. Bias Categorization

The authors categorize RM failures into two classes based on the Linear Representation Hypothesis:

• Low-Complexity Biases: Manifest as isolated linear directions in the RM's hidden representation space (e.g., length, uncertainty markers, position). These are amenable to linear intervention.
• High-Complexity Biases: Arise from entangled, context-dependent factors (e.g., sycophancy, model-style sensitivity) that cannot be approximated by a single linear direction.

B. Mechanistic Reward Shaping (The Intervention)

For low-complexity biases, the paper proposes a post-hoc, model-internal intervention that does not require retraining the RM or modifying the policy optimization loop. The process involves:

1. Probe Construction: Using the Difference-of-Means (DiffMean) method to construct linear activation probes. These probes identify directions in the representation space that correlate with specific biases (e.g., the vector difference between "verbose correct" and "concise correct" responses).
2. Null-Space Projection: Once a probe direction $p$ is identified, the authors project the RM's hidden activation $h$ onto the orthogonal complement of $p$ to remove the bias component:
   $$h_{null} = h - \alpha (p^\top h) p$$
   Where $\alpha$ controls the projection strength. This effectively "nuls" the bias signal from the activation before the reward head computes the final score.

C. Evaluation Setup

• Models: Five RMs were evaluated, including SoTA models from the Skywork family (Llama-3.1 and Qwen3 based) and older models (DeBERTa, AllenAI).
• Datasets: Four diverse benchmarks were used for bias detection: PlausibleQA, BIG-bench, GSM8K-MC, and MMLU.
• Generalization: RewardBench-2 was used to ensure interventions did not degrade general ranking performance (Out-of-Distribution testing).

3. Key Contributions & Results

A. Persistent and Novel Biases

The systematic evaluation of five RMs revealed:

• Length Bias: SoTA models trained to avoid verbosity often overcorrect, penalizing long correct answers and preferring concise incorrect ones. DeBERTa models exhibited classic length bias (preferring long answers).
• Uncertainty Bias: RMs systematically penalize expressions of uncertainty (e.g., "I'm not sure"), even when the answer is correct. This leads to overconfidence.
• Position Bias: RMs show significant bias toward answers based on their position in a list (e.g., preferring the first or last option), affecting both multiple-choice and free-form text settings.
• Model-Style Sensitivity: A novel finding where RMs correlate rewards with the stylistic "fingerprint" of specific generative models (measured via cross-entropy), effectively rewarding outputs that look like they came from the model's own training distribution.

B. Effectiveness of Mechanistic Reward Shaping

The proposed linear probe intervention successfully mitigated low-complexity biases:

• Length: The intervention closed the accuracy gap between concise and verbose correct answers without degrading baseline performance.
• Uncertainty: The method reduced the penalty for uncertainty, increasing the RM's preference for uncertain-but-correct answers while maintaining preference for direct answers when the model is confident.
• Position: Nulling position probes significantly reduced variance in accuracy across different answer positions (A, B, C, D).
• Calibration: By removing spurious confidence signals, the method improved the Spearman correlation between verbalized confidence and actual correctness (calibration), particularly for Skywork-Qwen models.

C. Limitations and High-Complexity Biases

• Sycophancy (User Agreement): The intervention failed to mitigate sycophancy. The authors found that "agreeing with the user" is co-linear with "being helpful" in the activation space. Removing the sycophancy signal also removed helpful agreement, indicating this is a high-complexity, entangled bias.
• Model-Style Sensitivity: Similarly, style sensitivity could not be removed via linear probes because the stylistic features are entangled with content quality.

D. Generalization and Safety

• Out-of-Distribution (OOD): Probes trained on specific datasets (e.g., GSM8K for length) generalized effectively to RewardBench-2.
• Performance Preservation: The interventions did not significantly degrade the RM's ability to rank chosen vs. rejected completions on RewardBench-2 (statistically non-inferior to baselines).
• Reward Distribution: The interventions narrowed the reward gap between chosen and rejected samples, suggesting the removal of artificial inflation/deflation caused by surface features.

4. Significance

• Paradigm Shift: The paper moves beyond treating reward hacking solely as a data-augmentation or retraining problem. It introduces a mechanistic, interpretability-based approach to fix RMs post-hoc.
• Efficiency: The method is data-efficient (requires only a small probe dataset) and computationally cheap (linear projection), making it applicable to already-deployed RMs without retraining.
• Diagnostic Framework: By distinguishing between linear (tractable) and non-linear (entangled) biases, the paper provides a roadmap for future alignment research, clarifying which biases can be "fixed" with current tools and which require fundamental architectural or training changes.
• Safety Implications: The findings on sycophancy and style sensitivity highlight that even "aligned" models may harbor deep, structural biases that simple linear fixes cannot resolve, posing risks for high-stakes applications like mental health or clinical decision-making.

Conclusion

The paper demonstrates that while simple, linear biases in Reward Models can be effectively mitigated through mechanistic reward shaping (null-space projection), complex, entangled biases like sycophancy and model-style sensitivity remain persistent challenges. This work underscores the need for a nuanced understanding of RM failure modes, distinguishing between those that are artifacts of spurious correlations and those that are intrinsic to the model's learned representations.