🤖 AI

Mitigating Cognitive Bias in RLHF by Altering Rationality

This paper proposes a method to mitigate cognitive bias in Reinforcement Learning from Human Feedback (RLHF) by dynamically adjusting the rationality parameter during reward learning, using an LLM-as-judge to identify and downweight biased human comparisons, thereby training more robust models even on imperfect datasets.

Original authors: Tiffany Horter, Andrew Markham, Niki Trigoni, Serena Booth

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tiffany Horter, Andrew Markham, Niki Trigoni, Serena Booth

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to make good decisions. The standard way to do this is called RLHF (Reinforcement Learning from Human Feedback). You show the robot two different answers to a question, ask a human which one is better, and the robot learns to mimic that choice.

The problem, according to this paper, is that humans aren't perfect teachers. Sometimes, we make mistakes because of our brains' built-in shortcuts, known as cognitive biases.

Think of cognitive biases like a pair of distorted glasses. When you wear them, you might see the world clearly, but sometimes you see things that aren't really there, or you miss important details. A famous example is the "Linda Problem": if you tell someone Linda is a feminist philosophy major, they might irrationally think she is more likely to be a "feminist bank teller" than just a "bank teller," even though mathematically, the second option must be more probable. If a human wears these "distorted glasses" while teaching the robot, the robot learns to wear them too.

The Old Way vs. The New Way

The Old Way (Fixed Glasses):
In traditional training, the robot assumes the human teacher is equally reliable for every single question. It treats every "I like this answer" vote with the same weight, like a teacher giving every student the same grade regardless of whether they were paying attention or daydreaming. The researchers call this a fixed "rationality parameter" (let's call it Beta). It's like assuming the human teacher is always 100% sober and focused.

The New Way (Dynamic Glasses):
The authors, Tiffany Horter and her team, propose a smarter approach. They suggest that the robot should ask: "Is this human teacher likely to be biased right now?"

They built a system that acts like a spot-check supervisor. Before the robot learns from a human's choice, a second AI (an "LLM-as-judge") looks at the question and the human's choice to see if it smells like a cognitive bias.

If the supervisor thinks, "Ah, this looks like a classic bias trap," it tells the robot to turn down the volume on that human's feedback. It lowers the "Beta" value, making that specific lesson less important.
If the supervisor thinks, "This looks like a solid, rational choice," it turns the volume up, letting the robot learn heavily from it.

The Analogy: The Noisy Classroom

Imagine a classroom where the teacher is trying to teach students (the AI) math.

The Problem: Sometimes the teacher is tired, distracted, or falling for a trick question. If the students copy the teacher's mistakes, they fail the test.
The Old Method: The students copy everything the teacher writes on the board, assuming the teacher is always right.
The New Method: The students have a smart assistant sitting next to them. When the teacher writes an answer, the assistant whispers, "Hey, I think the teacher is making a mistake here because of a trick question." The students then ignore that specific answer and focus on the ones where the teacher seems sharp.

What Happened When They Tried It?

The researchers tested this on two sets of tricky questions designed to trip up human logic (like the "Linda" problem and medical diagnosis scenarios).

It stopped the robot from copying mistakes: Even when the robot was trained on data where humans were mostly wrong (biased), the new method helped the robot figure out the right answer. It learned to ignore the "distorted glasses" and find the truth.
It didn't break the robot: The robot didn't get confused or forget how to do other things. It stayed smart at general tasks while just getting better at ignoring bias.
It worked even with a "flawed" supervisor: The system didn't need a perfect supervisor to spot every single bias. Even if the supervisor was only right 80% of the time, the robot still learned better than if it had no supervisor at all.

The Big Takeaway

The paper argues that we shouldn't treat human feedback as a perfect, unchangeable signal. Instead, we should treat it like a noisy signal that changes depending on the context. By dynamically adjusting how much weight we give to human feedback based on how likely it is to be biased, we can build AI that is more rational and less likely to inherit our human flaws.

In short: Don't just listen to what humans say; listen to how they say it, and adjust your learning accordingly.

Technical Summary: Mitigating Cognitive Bias in RLHF by Altering Rationality

Problem Statement

Reinforcement Learning from Human Feedback (RLHF) relies on the premise that human annotators express preferences that faithfully reflect their latent intentions. However, this premise is flawed because human judgment is frequently subject to cognitive biases (e.g., the conjunction fallacy, anchoring bias, sunk cost fallacy). In standard RLHF pipelines, these systematic deviations are treated as uniform stochastic noise. The learning process typically models human preferences using a Boltzmann-rational formulation where a fixed rationality parameter, $\beta$ , governs the consistency between latent reward differences and observed pairwise preferences.

Treating $\beta$ as a global constant implicitly assumes uniform annotator reliability across all contexts. Consequently, when training data contains systematic cognitive biases, the learned reward model embeds these biases, causing downstream Large Language Models (LLMs) to reproduce or even amplify irrational behaviors. The core problem addressed is how to make RLHF robust to imperfect human feedback where deviations from rationality are not random but context-dependent and systematic.

Methodology

The authors propose a framework that treats rationality as context- and annotation-dependent rather than fixed. Instead of a global $\beta$ , they introduce a dynamic, instance-dependent rationality parameter, $\beta_{new}$ , which modulates the weight of each preference pair during reward learning based on the estimated likelihood of cognitive bias.

1. Dynamic Rationality Parameter ( $\beta_{new}$ )

The method replaces the fixed $\beta$ with a function of the estimated bias probability ( $D_f$ ) for a specific comparison:
$\beta_{new} = \text{logistic}(k \cdot (\theta - D_f))$
Where:

$D_f \in [0, 1]$ is the estimated probability that the observed feedback is influenced by cognitive bias.
$\theta$ is a threshold parameter determining when feedback begins to be downweighted.
$k$ controls the steepness of the transition.

When $D_f$ is low (low bias likelihood), $\beta_{new}$ approaches 1, placing high weight on the preference. As $D_f$ increases, $\beta_{new}$ decreases toward 0, effectively downweighting the influence of potentially biased judgments on the reward model.

2. Bias Detection ( $D_f$ Estimation)

To estimate $D_f$ , the authors employ an "LLM-as-judge" approach. An out-of-the-box LLM (e.g., ChatGPT 5.2 Thinking or Mistral-7B-Instruct) evaluates each training example (prompt, candidate responses, and the chosen preference) to predict the likelihood of bias.

Pairwise Evaluation: The judge identifies which of two candidate responses is more biased.
Calibration: A lightweight calibration step corrects for systematic directionality errors across different bias types using a minimal held-out validation set.
Granularity: Crucially, $D_f$ is defined with respect to the chosen response within a specific scenario, not the scenario itself. This allows the system to retain informative signals from bias-inducing scenarios where the annotator successfully avoids the bias (e.g., a statistically trained annotator correctly rejecting the conjunction fallacy).

3. Training Pipeline

Reward Model: A reward model (initialized from Mistral-7B) is trained using a loss function where the weight of each pair is conditional on $\beta_{new}$ .
Policy Optimization: The downstream policy is finetuned using Group Relative Policy Optimization (GRPO) with the frozen, bias-aware reward model.
Datasets: The method is validated on two datasets designed to elicit cognitive biases: the BRU Dataset (205 questions, 8 bias types) and the CogBias Dataset (30,000 cases, 30 bias types). To simulate realistic RLHF failure modes, the training data is constructed such that biased responses are preferred over ground-truth responses in a tunable fraction of cases (e.g., 3:1 ratio).

Key Contributions

Context-Dependent Rationality: The paper challenges the standard assumption of a fixed rationality parameter in RLHF, proposing instead that $\beta$ should vary based on the reliability of specific feedback instances.
Dynamic Downweighting Mechanism: The authors introduce a mechanism to dynamically adjust $\beta$ using an LLM-based bias detector, allowing the system to downweight biased comparisons while preserving informative signals from unbiased judgments within the same context.
Robustness to Imperfect Detection: The method demonstrates effectiveness even when the bias detector (the LLM-as-judge) is imperfect or noisy, provided the detector's accuracy remains above a certain threshold (approx. 57-60%).

Experimental Results

The authors evaluated their approach on the BRU and CogBias datasets, comparing the dynamic $\beta$ method against baselines using fixed $\beta$ values (0.1, 0.5, 0.9, 1.0) and random $\beta$ assignments.

Reduction in Bias Propagation: On the CogBias dataset, the debiased model achieved a ground-truth accuracy of 67.6%, a significant improvement over the un-finetuned base model (42.4%) and baselines trained on the same biased data (which collapsed to ~1.7% accuracy). On the BRU dataset, the debiased model improved accuracy by 8.3% over the base model despite being trained on 3:1 biased data.
Robustness to Biased Feedback: The method successfully prevented the "collapse toward biased preferences" observed in baselines. Even when training data contained a 5:1 ratio of biased-to-ground-truth preferences, the dynamic $\beta$ model maintained significantly higher accuracy than fixed- $\beta$ baselines.
Preservation of General Performance: Evaluation on the TruthfulQA dataset showed no significant degradation in general performance (non-significant change of -0.5%), indicating the intervention selectively reduces bias without sacrificing overall capability.
Cross-Architecture and Transferability: The approach yielded statistically significant improvements across different model architectures (Mistral-7B and Qwen3-8B). Furthermore, a model finetuned on the small BRU dataset successfully transferred its ability to mitigate anchoring bias to the larger CogBias dataset, improving ground-truth accuracy from 37.7% to 44.3%.

Significance and Claims

The paper argues that effective alignment in RLHF depends less on obtaining idealized, perfectly rational human feedback and more on modeling the distortions inherent in that feedback. By reframing preference data as context-dependent and systematically biased, the authors demonstrate that reward models can be trained to leverage imperfect feedback without collapsing into its biases.

The authors claim that their work contributes to a growing direction in alignment research: moving away from the assumption of uniform rationality toward methods that detect, model, and adapt to cognitive biases. They posit that treating $\beta$ as a signal of feedback fidelity, rather than a fixed constant, allows LLMs to approximate deliberative reasoning rather than inheriting human heuristics. The paper concludes that while the success of the method is partially dependent on the accuracy of the bias detector, the intervention remains robust even with imperfect judges, suggesting that robustness to human inconsistency does not require perfect models of cognition.