Making Bias Non-Predictive: Training Robust LLM Reasoning via Reinforcement Learning

1. Problem Statement

Large Language Models (LLMs), particularly those used as automated evaluators ("LLM-as-a-Judge") or reasoners, are highly susceptible to cognitive biases. These biases cause models to alter their reasoning and final decisions based on spurious, non-causal prompt-level cues (e.g., "90% of people agree," "Experts recommend," or option positioning) rather than factual correctness.

• The Core Issue: Existing mitigation strategies (prompt engineering, Supervised Fine-Tuning/SFT) fail because they only modify surface behavior without changing the underlying optimization objective. During pretraining, bias cues often correlate with positive feedback, making them attractive "shortcuts" for the model.
• The Consequence: Even models with strong reasoning capabilities (like OpenAI-o1 or DeepSeek-R1) can be manipulated to reverse correct judgments when exposed to fabricated consensus or authority claims. This represents a failure of Epistemic Independence: the ability to make judgments invariant to spurious signals.

2. Methodology: Epistemic Independence Training (EIT)

The authors propose Epistemic Independence Training (EIT), a Reinforcement Learning (RL) framework grounded in a single principle: To learn independence, bias cues must be made non-predictive of reward. If a bias signal predicts correctness, an optimal policy will exploit it; if it provides no directional advantage, the model is forced to rely on intrinsic reasoning.

EIT consists of three key components:

A. Balanced Conflict Strategy (Data Construction)

Instead of training only on "bias is wrong" examples (which teaches the model to simply oppose bias), EIT uses a 50/50 conflict strategy:

• In 50% of training samples, the injected bias signal supports the correct answer.
• In 50% of samples, the bias signal supports the incorrect answer.
• Result: The correlation between the bias cue and the correct answer is zero ($P(\text{bias} = \text{correct}) = 0.5$). This renders the bias cue statistically non-predictive, forcing the model to ignore it to maximize reward.

B. Hierarchical Reward Design

The reward function $R$ is composed of three parts to prevent specific failure modes:

1. Structural Reward ($R_{struct}$): Ensures the model outputs a valid Chain-of-Thought (CoT) format.
2. Factual Accuracy ($R_{acc}$): Rewards correct answers to prevent "random contrarianism" (guessing the opposite of the bias without reasoning).
3. Independence Incentive ($R_{ind}$): The critical component that decouples reasoning from social influence.
   • Adversarial Context (Bias $\neq$ Truth):
     • $+ \gamma_1$: Reward for ignoring bias and choosing the correct answer.
     • $- \gamma_1$: Penalty for following the bias to a wrong answer.
   • Supportive Context (Bias $=$ Truth):
     • $0$: No bonus for being correct when bias aligns with truth.
     • $- \gamma_2$: Penalty for ignoring the truth and following a wrong bias (if applicable).
   • Key Insight: By providing zero marginal gain when bias aligns with truth, the model cannot learn the heuristic "always follow bias." It must ignore the bias entirely to succeed in all scenarios.

C. Optimization

The framework utilizes Group Relative Policy Optimization (GRPO). The policy is updated to maximize the expected reward relative to a group baseline, reinforcing responses that demonstrate both high accuracy and independence.

3. Key Contributions

1. Formulation: Defines epistemic independence as an RL problem where spurious cues are rendered non-predictive of reward.
2. Novel Mechanism: Introduces the Balanced Conflict Strategy and Bias-Penalizing Reward, which jointly prevent shortcut learning and force reliance on task-relevant reasoning.
3. Generalization: Demonstrates that training on a single bias type (Bandwagon) induces transferable independence to unseen bias types (Authority, Distraction, Position) and across different benchmarks and model families.
4. Superiority over Scaling: Shows that targeted EIT training on a smaller model (Qwen3-4B) outperforms larger, untrained models (Qwen3-8B, Qwen3-14B) in bias resistance, proving that training quality matters more than parameter scale for robustness.

4. Experimental Results

Experiments were conducted on Qwen3-1.7B and Qwen3-4B using the MMLU-Pro dataset, with evaluation on MedQA and HellaSwag.

• Accuracy & Robustness Gains:
  • On Qwen3-4B, EIT improved adversarial bias accuracy by +13.2% (70.1% $\to$ 83.3%) and robustness by +16.4%.
  • Distraction Bias: Showed the strongest transfer, with wrong-bias accuracy jumping +39.1% (45.9% $\to$ 85.0%) despite never being seen during training.
• Comparison with Baselines:
  • vs. Prompt Engineering: Prompt-based mitigation often degraded clean accuracy (e.g., Qwen3-4B clean accuracy dropped from 66.2% to 40.9%).
  • vs. SFT: SFT led to "performative independence" (models saying they are independent but failing to reason) and overfitting. EIT achieved consistent gains across all settings.
  • vs. Larger Models: EIT-trained Qwen3-4B (83.3% accuracy under bandwagon bias) outperformed untrained Qwen3-8B (72.5%) and Qwen3-14B (75.2%).
• Cross-Benchmark & Cross-Model:
  • EIT generalized to unseen benchmarks (MedQA, HellaSwag) and model families (Llama-3.2-3B), confirming the method is not benchmark-specific.
  • It outperformed distribution-shift methods (GroupDRO, IRM) without requiring environment labels.

5. Significance and Qualitative Analysis

The paper provides a qualitative analysis of how models reason under bias, revealing three distinct patterns:

1. Baseline: Correct reasoning is derailed by self-doubt when encountering authority/consensus signals.
2. SFT: Exhibits "Performative Independence." The model learns the language of independence ("I should not let popular opinion influence me") but lacks the substance (actual reasoning), often resulting in wrong answers.
3. EIT: Exhibits "Genuine Epistemic Independence." The model engages domain knowledge, performs explicit verification (calculations, logic), and explicitly rejects conflicting bias signals based on its own reasoning.

Conclusion:
EIT demonstrates that robustness to cognitive bias is not an inherent property of model scale but a learnable capability. By making bias cues non-predictive through balanced conflict data and specific reward shaping, RL can instill genuine epistemic independence, making LLMs more reliable as automated judges and reasoners in adversarial environments.