Bias Fitting to Mitigate Length Bias of Reward Model in RLHF

Technical Summary: FiMi-RM (Bias Fitting to Mitigate Length Bias of Reward Model in RLHF)

Problem Statement

Reinforcement Learning from Human Feedback (RLHF) is the standard framework for aligning Large Language Models (LLMs) with human preferences. However, this process is susceptible to reward hacking, where the policy exploits flaws in the trained reward model to maximize scores without genuinely adhering to human intent. A prevalent form of this hacking is length bias, where reward models systematically favor longer responses regardless of their actual quality. This leads to excessively verbose generations in downstream models.

Existing mitigation strategies suffer from significant limitations:

1. Lack of Characterization: Some approaches (e.g., RRM) attempt to balance data distributions or use causal frameworks without explicitly modeling the form of the bias.
2. Linear Assumptions: Many methods (e.g., Length Penalty, ODIN, Huang et al.) assume a linear relationship between response length and reward. They either subtract a fixed coefficient based on length or use linear regression to decouple length from quality.
3. Inadequacy: These linear assumptions fail to capture the intricate, non-linear nature of how length interacts with reward scores, particularly the complex patterns observed in different length regimes.

Methodology: FiMi-RM

The authors propose FiMi-RM (Bias Fitting to Mitigate Length Bias of Reward Model), a framework designed to autonomously learn and correct underlying non-linear bias patterns. The approach operates in three distinct stages:

1. Warm-Up Stage:

   • A standard reward model ($model_r$) is trained using the Bradley-Terry loss on preference data.
   • Crucially, this stage preserves the inherent length bias of the reward model rather than attempting to correct it immediately. This ensures the model possesses a strong, systematic tendency to assign higher scores to longer responses, providing a clear target for the subsequent fitting stage.

2. Length Bias Fitting Stage:

   • A lightweight fitting model ($model_f$) is introduced to explicitly characterize the relationship between response length ($len(y)$) and the biased reward output ($r$).
   • Architecture: The scalar length is projected into a $d$-dimensional feature space (using Positional Encoding-inspired Length Encoding), processed through a ResNet architecture, and passed through a linear projection head to predict the reward ($\hat{r}$).
   • Objective: The fitting model is trained to minimize the discrepancy between its prediction ($\hat{r}$) and the actual reward model output ($r$). The optimization uses a Pearson correlation loss ($L_{pearson}$) to maximize the correlation between the predicted and actual rewards.
   • Key Insight: The use of Pearson loss aligns the correlation patterns without imposing a linearity constraint on the fitting model itself. The ResNet-based $model_f$ is free to learn complex, non-linear dependencies.

3. Length Debiasing Stage:

   • The original reward model is fine-tuned to decouple its outputs from response length while preserving its ability to model human preferences.
   • Training Mechanism: The framework employs an alternating training strategy between the reward model and the fitting model.
   • Loss Function: The reward model is optimized using a composite loss:
     $$L_{debiased} = L'_{pearson} + L_{BT}$$
     • $L'_{pearson}$: Forces the reward model's output to be uncorrelated with the fitting model's prediction (effectively removing the learned bias).
     • $L_{BT}$: The standard Bradley-Terry loss, ensuring the model retains its discriminative capability for human preferences.
   • The indicator function $I(step)$ alternates training steps between fitting the bias and debiasing the reward model.

Key Contributions

1. Non-Linear Bias Modeling: The paper introduces a multi-stage framework that autonomously learns the non-linear relationship between response length and "hacked" reward, moving beyond the simplistic linear assumptions of prior work.
2. Empirical Validation of Non-Linearity: Through the fitting process, the authors identify a multi-stage bias pattern:
   • Short responses (<100 tokens): Exhibit a strong linear correlation where reward increases with length.
   • Longer responses: The relationship flattens, and in some cases, shows a slight downward tendency, indicating that the positive effect of length on reward diminishes or reverses for extended outputs.
3. Comprehensive Evaluation: The method is validated through length-reward distribution analysis, length-controlled win rates, and response length distributions across multiple alignment algorithms (DPO and Best-of-N).

Experimental Results

The authors evaluated FiMi-RM using Qwen2.5-7B and Gemma2-9B models on the Anthropic HH dataset, comparing against Vanilla Reward Models, Length Penalty, and ODIN.

• Preference Accuracy: FiMi-RM achieved a more balanced accuracy across subsets where the chosen response was longer (C-longer) versus shorter (R-longer). While overall accuracy on the C-longer subset decreased slightly, the authors argue this indicates the successful removal of the "length shortcut" rather than a loss of semantic understanding.
• Length-Reward Distribution: Scatter plots revealed that FiMi-RM produces a more symmetric and flatter reward distribution across length bins compared to baselines, confirming effective debiasing.
• Downstream Performance (BoN & DPO):
  • Win Rates: FiMi-RM achieved the highest Length-Controlled Win Rate (LC-WR) and standard Win Rate (WR) in both Best-of-N (BoN) and Direct Preference Optimization (DPO) settings.
  • Verbosity: The method significantly reduced the average token length of generated responses compared to Vanilla RM and Length Penalty, while maintaining or improving performance on benchmarks like MT-Bench and IFEval.
  • Selection Bias: In BoN selection, FiMi-RM demonstrated a stronger preference for concise outputs compared to ODIN, which tended to shift preferences toward medium-length responses rather than short ones.

Significance and Claims

The paper claims that FiMi-RM offers a more precise and effective approach to mitigating length bias by explicitly modeling the non-linear nature of the problem. By decoupling length from reward without compromising the core preference modeling capabilities, the method enables RLHF pipelines to generate responses that are both high-quality and appropriately concise.

The authors note that while their method effectively reduces spurious correlations, the question of whether human preferences are entirely independent of length remains open. They acknowledge that in certain contexts (e.g., tasks requiring detailed explanations), a positive correlation between length and quality may be genuine. However, their framework successfully distinguishes between exploitable bias and genuine preference, leading to safer and more effective model alignment.