Technical Summary: Diagnosing and Mitigating Systemic Reward Bias in Self-Rewarding RL

1. Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for scaling the reasoning capabilities of Large Language Models (LLMs) but is constrained by the scarcity of labeled data. Reinforcement Learning with Intrinsic Rewards (RLIR), or self-rewarding, offers a scalable alternative by allowing the policy model to generate its own reward signals. However, RLIR methods often suffer from instability and inferior performance compared to RLVR.

The authors identify the root cause of this gap as a systemic reward bias inherent in confidence-coupled self-rewarding. In existing RLIR frameworks, reward estimation is tightly coupled with the policy's own confidence. This creates an asymmetric error pattern:

• Confident Correctness: Reward errors remain small.
• Confident Mistakes: Reward errors become large, leading to over-rewarding (assigning high rewards to incorrect outputs that the model is confident about).

This asymmetry forms a self-confirming loop where biased rewards accumulate during training, driving the policy toward over-rewarding, causing optimization instability, and capping the achievable performance ceiling.

2. Methodology: RLER (Reinforcement Learning with Ensembled Rewards)

To mitigate this systemic bias, the authors propose RLER, a framework that aggregates diverse policies to construct a unified reward-estimation space. RLER is designed to satisfy three desiderata: Accuracy (low noise), Unbiasedness (no directional drift), and Robustness (decoupling from policy confidence).

RLER achieves this through three core mechanisms:

A. Ensemble-Based Unified Rewarding

Instead of a single policy rewarding itself, RLER employs an ensemble of $K$ diverse source policies.

• Aggregation: Rollouts are sampled from all policies, and their answer distributions are pooled to form a unified ensemble distribution $\bar{p}_j$.
• Majority Consensus: A unified majority label $m_{EC}$ is derived from the ensemble distribution rather than a single model's output.
• Benefit: This reduces reliance on any single policy's confidence, thereby lowering reward noise ($\rho_{noise}$) and policy-reward coupling ($\rho_{selfbias}$).

B. Adaptive Soft-Reward Interpolation

To balance the high variance of hard rewards (binary 0/1) and the low-confidence bias of soft rewards (probabilistic), RLER introduces an adaptive interpolation strategy.

• Mechanism: The final reward $r_i^{(\alpha)}$ is a weighted blend of a hard ensemble decision ($r^H$) and a soft unified probability ($r^S$):
  $$r_i^{(\alpha)} = (1 - \alpha)r^S_i + \alpha r^H_i$$
• Adaptive Gate: The weight $\alpha(x)$ is dynamically modulated by the ensemble's unified confidence. High confidence increases reliance on the hard reward, while lower confidence increases reliance on the soft reward to preserve information about plausible minority answers.
• Calibration: The method incorporates token-level confidence and batch-wise min-max normalization to ensure robustness against varying difficulty levels.

C. Disagreement-Aware Rollout Selection

This mechanism explicitly targets the over-reward skew ($\rho_{symbias}$) by filtering updates based on ensemble disagreement.

• Strategy: The system allocates an update budget based on the reliability of the consensus.
  • High Consensus (Majority): Updates are scaled by $\alpha(x)$, emphasizing updates only when the ensemble is reliable.
  • Disagreement (Tail): Updates rely on the soft term $(1-\alpha)r^S$. This prevents the aggressive suppression of minority answers that might be correct, while naturally attenuating low-frequency noise.
• Outcome: High-confidence false positives (FPs) are aggressively filtered, while false negatives (FNs) are recovered, correcting the drift toward over-reward.

D. Ensemble-to-Single Consolidation

To ensure practical deployment without additional inference costs, the $K$ trained policies are merged into a single model using Ties-Merging. This allows the system to leverage ensemble benefits during training while deploying a single efficient policy.

3. Key Contributions and Diagnostics

The paper introduces a diagnostic framework to quantify the failure modes of RLIR using three metrics:

1. Reward Noise Magnitude ($\rho_{noise}$): Measures the absolute deviation of reward estimates from the oracle.
2. Policy-Reward Coupling ($\rho_{selfbias}$): Measures how tightly reward estimates are coupled to the policy's own confidence.
3. Symmetry Bias Rate ($\rho_{symbias}$): Measures the directional skew between over-reward (FP) and under-reward (FN).

Key Findings from Controlled Experiments:

• Over-reward is more detrimental than under-reward: Experiments show that FP-dominated noise causes a more severe performance drop than FN-dominated noise.
• Coupling amplifies errors: High $\rho_{selfbias}$ reinforces confidence-conditioned errors, leading to unstable training dynamics.
• Decoupling is essential: Reducing the coupling between the policy and its reward signal is critical for stability.

4. Experimental Results

Extensive experiments were conducted on math-style reasoning tasks (Arithmetic, DAPO-MATH-17K) and cross-model/dataset generalization (LLaMA, Qwen, Big-Math, WebInstruct).

• Performance: RLER improves by +6.2% over the best RLIR baseline (e.g., Co-rewarding, INTUITOR) and achieves performance within 3.6% of the RLVR upper bound (oracle-labeled).
• Stability: RLER exhibits smooth, consistent scaling as the training set size increases (from 8k to 1M samples), whereas standard RLIR baselines show unstable scaling and performance plateaus.
• Metric Improvement: RLER significantly reduces $\rho_{noise}$, $\rho_{selfbias}$, and $\rho_{symbias}$, effectively breaking the self-confirming loop.
• Ablation Studies: Removing any component (Ensemble, Interpolation, or Selection) leads to performance degradation, confirming that all three mechanisms contribute cumulatively to the bias mitigation.

5. Significance and Claims

The paper claims that RLER provides a practical, scalable solution for reinforcement learning in unlabeled settings where human supervision is scarce. By diagnosing the systemic bias of self-rewarding and addressing it through ensemble aggregation and adaptive selection, RLER:

• Enables stable scaling on large unlabeled corpora without the need for external reward models.
• Bridges the performance gap between self-rewarding (RLIR) and oracle-labeled (RLVR) methods.
• Offers a deployment-friendly approach via model merging, incurring no additional inference overhead.

The authors conclude that RLER represents a significant step toward sustainable, self-improving reasoning systems for LLMs, particularly in domains with abundant unlabeled data but scarce ground truth.