💬 NLP

Understanding helpfulness and harmless tension in reward models

This paper investigates the internal mechanisms of reward models in RLHF, revealing that interference between helpfulness and harmlessness objectives arises from shared neurons that causally support one goal while undermining the other, thereby explaining why mixed-objective models often underperform single-objective ones.

Original authors: Eshaan Tanwar, Pepa Atanasova

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eshaan Tanwar, Pepa Atanasova

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Tug-of-War" Inside AI Brains

Imagine you are training a new employee (an AI) to be the perfect assistant. You have two main rules for them:

Be Helpful: Answer every question, solve every problem, and be super useful.
Be Harmless: Never say anything dangerous, offensive, or private.

The problem is that these two rules sometimes fight each other. If a user asks for a "private email address," being helpful means giving the answer, but being harmless means refusing because it's a privacy violation.

This paper investigates what happens inside the "brain" of the AI's Reward Model (the part that decides which answers are good) when we try to teach it both rules at the same time. The authors found that trying to do both at once often makes the AI worse at both, and they figured out why by looking at the specific "neurons" (tiny switches) inside the model.

1. The Experiment: Three Different Coaches

To understand the conflict, the researchers trained three different versions of the AI's "judge" (Reward Model):

The "Helpful" Coach: Only trained to pick the most useful answers.
The "Harmless" Coach: Only trained to pick the safest answers.
The "Mixed" Coach: Trained to be both helpful and harmless at the same time.

The Result: The "Mixed" Coach was actually worse than the other two. It couldn't decide what to do as well as the specialists. It was like a coach trying to teach a soccer player to be the world's best striker and the world's best goalie at the same time; the player ended up being mediocre at both.

2. The "Shared Neurons" Problem

The researchers wanted to know why the Mixed Coach struggled. They looked inside the AI's brain to see which "neurons" (tiny processing units) were firing when the AI thought about being helpful versus being harmless.

The Discovery: They found that about half of the neurons used for being helpful were the exact same neurons used for being harmless.

The Analogy: Imagine a busy kitchen.

The "Helpful" chef uses a specific set of knives to chop vegetables quickly.
The "Harmless" chef uses a different set of knives to carefully avoid cutting fingers.
The researchers found that both chefs are trying to use the same 50% of the knives.

When the "Mixed" chef tries to use those shared knives, they get confused. If they chop too fast to be helpful, they might cut a finger (harmless). If they move too slowly to be safe, they don't get the food done (helpful). Because the same "knives" (neurons) are being pulled in two different directions, the whole system gets jammed.

3. The "Interference" Effect

The paper shows that these shared neurons are incredibly powerful.

When the researchers "turned off" (ablated) the neurons specific to being helpful, the AI got worse at being helpful but actually got better at being harmless.
When they turned off the harmless neurons, the AI got worse at being harmless but better at being helpful.

This proves that these neurons aren't just passive helpers; they are actively fighting each other. In the "Mixed" model, the helpful neurons are constantly trying to push the AI to answer, while the harmless neurons are trying to push it to refuse. Because they share the same hardware, they cancel each other out, leading to a confused AI that scores lower on both tests.

4. The "Weak Signal"

Even when the Mixed model managed to make the right choice (like refusing a bad request), the researchers noticed something else. The "confidence score" it gave itself was much lower than the specialist models.

The Analogy:

The Helpful Coach says: "Yes, this answer is great! I'm 100% sure!"
The Harmless Coach says: "No, this is bad! I'm 100% sure!"
The Mixed Coach says: "Um, maybe this is okay? I'm... 60% sure?"

The Mixed model is less confident because the internal conflict makes it hesitate. It's like a person trying to walk in two different directions at once; they end up shuffling forward slowly and unsurely.

Summary of Findings

Specialists win: An AI trained to be just helpful or just harmless performs better than one trained to be both.
Shared hardware causes conflict: The reason for the failure isn't just a lack of data; it's that the AI uses the same internal parts (neurons) for both tasks, and those parts get confused when asked to do opposing things.
The solution isn't simple: You can't just "mix" the training data and expect a perfect result. The paper suggests we need to find ways to separate these "shared neurons" so the AI can be helpful without fighting its own safety mechanisms.

In short, the paper reveals that the struggle between being helpful and being safe isn't just a rulebook problem; it's a physical wiring problem inside the AI's brain where the same switches are trying to do two opposite jobs at once.

Technical Summary: Understanding Helpfulness and Harmless Tension in Reward Models

Problem Statement

Reward Models (RMs) are central to Reinforcement Learning from Human Feedback (RLHF), tasked with aligning Large Language Models (LLMs) with human values, specifically helpfulness and harmlessness. While these objectives are often pursued simultaneously in mixed-objective training settings, they frequently conflict. For instance, an RM might prioritize refusing a query to ensure harmlessness (e.g., refusing to provide a fictional character's private email) when a human prefers a helpful, informative response. This phenomenon, termed alignment tension, leads to suboptimal model behavior where an aligned LLM may either produce unsafe responses to satisfy helpfulness or refuse benign queries to ensure harmlessness.

Despite the prevalence of this issue, the internal mechanisms of RMs regarding how they encode, represent, and reconcile these competing objectives remain poorly understood. Existing interpretability work largely focuses on explaining specific scoring decisions rather than analyzing how competing objectives interact within the model's internal circuitry.

Methodology

The authors investigate alignment tension through a combination of behavioral analysis and mechanistic interpretability.

Experimental Setup

Model Training: The study trains three variants of Reward Models from the same pre-trained LLM backbones (SmoLLM2 and GPT-2 families):
- Helpful RM ( $r^H_\theta$ ): Trained exclusively on the helpfulness subset of the HH-RLHF dataset.
- Harmless RM ( $r^S_\theta$ ): Trained exclusively on the harmlessness subset.
- Mixed RM ( $r^{HS}_\theta$ ): Trained on the combined dataset containing both objectives.
Evaluation: The models are evaluated on diverse benchmarks (RewardBench, RewardBench2, RM-Bench) categorized into helpfulness-oriented tasks (e.g., chat, reasoning, instruction following) and harmlessness-oriented tasks (e.g., safety, refusal, adversarial robustness).
Neuron Analysis:
- Identification: The authors identify "oriented neurons" by calculating magnitude-based importance scores. Neurons with activations in the top $\tau$ -th percentile (set to 10) on specific objective datasets are classified as helpfulness-oriented or harmlessness-oriented.
- Causal Intervention: Targeted ablation is performed by setting the activations of these identified neurons to zero during the forward pass to measure their causal impact on model performance.
- Overlap Analysis: The study quantifies the overlap between neuron sets identified in single-objective models versus the mixed-objective model.

Key Results

1. Behavioral Performance and Tension

Underperformance of Mixed Models: Mixed-objective RMs consistently underperform single-objective RMs across diverse evaluation tasks. They fail to match the specialized performance of models trained on a single objective, indicating interference between objectives during joint optimization.
Asymmetry in Objectives: Single-objective models exhibit distinct behavioral tendencies. Helpful RMs excel at cooperative answering but may lack robustness against deceptive prompts. Harmless RMs show superior robustness in adversarial settings but may over-refuse benign queries.
Limited Retention: When behaviors learned by single-objective models are tested against the mixed-objective model, "behavioral retention" is relatively low (averaging ~56% for helpfulness and ~62% for harmlessness). While coarse behaviors (e.g., generic refusal) are retained, nuanced and adversarially robust behaviors degrade significantly.
Reward Calibration: Even when the mixed-objective model makes the correct decision (matching the single-objective model), its reward scores are consistently weaker and more concentrated near zero, suggesting a loss of confidence and signal separability.

2. Mechanistic Insights: Neuron Roles and Overlap

Causal Importance: Ablation studies confirm that neurons identified as specific to an objective are causally critical for that objective. Removing harmlessness-oriented neurons from a Harmless RM causes a significant drop in harmlessness task performance (approx. 12 points) but often improves helpfulness task performance, and vice versa.
Shared Representations: A substantial proportion of neurons (approximately 50%) associated with helpfulness and harmlessness are shared (i.e., highly activated in both single-objective models).
Disproportionate Influence: These shared neurons exert a disproportionate influence on model behavior. Ablating shared neurons in the mixed-objective RM causes a severe accuracy drop (10 points on harmlessness tasks, 4.4 points on helpfulness tasks), far exceeding the impact of ablating objective-specific (non-shared) neurons.
Layer Distribution: Oriented neurons are concentrated in the late layers of the model, which are known to capture high-level semantic features, suggesting that the conflict arises in the processing of complex semantic representations rather than surface-level syntax.

Key Contributions

Characterization of Alignment Tension: The paper demonstrates that joint training on helpfulness and harmlessness leads to performance degradation and reduced transferability compared to single-objective training, confirming that alignment tension is a representational interference problem.
Identification of Objective-Oriented Neurons: The authors identify specific neurons causally linked to alignment objectives, showing they are essential for their target objective but often detrimental to the opposing one.
Evidence of Shared Representations: The study reveals that nearly half of the neurons driving these objectives are shared. This overlap creates a bottleneck where optimizing one objective inherently conflicts with the other, driving the tension observed in mixed-objective models.

Significance and Claims

The paper claims that multi-objective alignment is not merely a behavioral trade-off but a representational interference problem arising from overlapping internal circuitry. The findings suggest that current mixed-objective training methods struggle because they force conflicting objectives to modify the same shared representations.

The authors argue that this mechanistic understanding is crucial for developing future disentangled and controllable alignment methods. By recognizing that shared neurons are the source of conflict, future work can focus on architectural or training strategies that separate these representations to mitigate alignment tension. The study provides a foundational step toward moving beyond black-box RLHF training toward more principled, interpretable reward modeling.

Limitations

The authors note that their analysis is limited to the reward modeling stage of RLHF and does not extend to policy optimization stages like Direct Preference Optimization (DPO). Additionally, computational constraints prevented the use of parameter-efficient fine-tuning, and the study focuses on broad definitions of helpfulness and harmlessness rather than fine-grained distinctions found in newer datasets.