Technical Summary: BiasGRPO

Problem Statement

Mitigating social bias in Large Language Models (LLMs) presents a unique alignment challenge distinct from verifiable tasks like math or coding. Bias lacks a single ground truth, resulting in a high-variance, subjective reward landscape. Existing preference-based fine-tuning methods face significant trade-offs in this domain:

• Direct Preference Optimization (DPO): As an offline algorithm relying on static pairwise instances, DPO suffers from limited exploration and lower generalization.
• Proximal Policy Optimization (PPO): While an online method that allows for better exploration, PPO relies on a separate critic (value) model. In noisy, subjective reward landscapes, these critic estimates can be unreliable, leading to training instability and high-variance updates.

There is currently no preference-based technique that successfully balances the stability of offline methods with the generalization benefits of online exploration specifically for bias mitigation.

Methodology: BiasGRPO

The authors propose BiasGRPO, a framework adapting Group Relative Policy Optimization (GRPO) to the specific constraints of bias mitigation. The framework consists of three core components:

1. The Algorithmic Core: Group-Relative Optimization

BiasGRPO replaces the traditional critic model used in PPO with a group-relative baseline.

• Mechanism: For each prompt, the policy model generates a group of completions (e.g., $G=4$). Instead of comparing a single completion to a critic's value estimate, the advantage is calculated by normalizing the reward of each completion against the mean and standard deviation of the entire group.
• Advantage Calculation:
  $$\hat{A}_{i,t} = \frac{r_i - \text{mean}(r)}{\text{std}(r)}$$
  Where $r_i$ is the reward for the $i$-th completion and $r$ is the set of rewards for the group.
• Rationale for Bias: In subjective domains, a "perfect" unbiased response is difficult to define. However, by generating a group of variations, the model can learn to identify which outputs are relatively less biased and more respectful, even if all outputs contain some degree of bias. This relative normalization provides a clearer, more direct training signal than the noisy advantage estimates derived from a single critic in high-variance landscapes.

2. Synthetic Dataset Extension

To support comprehensive training across diverse contexts, the authors curated a dataset of 20,999 prompts spanning 11 domains. This dataset integrates:

• BiasDPO: Extended synthetically to include 8,855 additional entries.
• Civil Comments: Real-world social media posts to capture bias elicited from neutral and biased inputs.
• UnQover: Ambiguous scenarios designed to elicit biased responses.
  The dataset includes corresponding favorable and unfavorable completions for DPO comparison and prompts for online methods.

3. Custom Bias Reward Model

The authors developed a specialized, compute-efficient reward model (0.1B parameters) to guide generation without causing knowledge degradation.

• Construction: A dataset of 2,930 unique sentences from StereoSet, CrowS-Pairs, and BBQ was resampled into 9,000 sets.
• Annotation: Using Best-Worst Scaling (BWS) and Iterative Luce Spectral Ranking (ILSR), a rotating cycle of LLMs (GPT-4o, Gemini 2.0 Flash, Claude 4 Sonnet) annotated bias severity. This multi-LLM approach mitigated the risk of circularity and inherited bias.
• Training: A RoBERTa model was trained to predict reward values (scaled [0, 1]), achieving a final MSE of 0.006.
• Validation: The model achieved the highest Spearman's rank correlation (0.4748) with Perspective API "identity attack" scores compared to existing bias scoring models.

Experimental Setup

• Base Model: Microsoft's 2.7B Phi-2 model, chosen as a "clean slate" testbed (no prior RLHF or bias mitigation).
• Baselines: The authors compared BiasGRPO against DPO (using the Identity Preference Optimization variant) and PPO.
• Benchmarks:
  • BOLD: Measuring representational harm (lower is better).
  • RealToxicityPrompts: Measuring overt hostility (lower is better).
  • BBQ: Measuring implicit stereotyping in ambiguous scenarios (higher is better).
  • TruthfulQA: Evaluating knowledge retention to ensure no catastrophic forgetting (higher is better).

Key Results

BiasGRPO outperformed both DPO and PPO across all benchmarks:

• Bias Reduction: The GRPO model achieved the lowest scores on BOLD (0.0140 vs. 0.0222 for DPO and 0.0268 for PPO) and RealToxicityPrompts (0.0198 vs. 0.0234 for DPO).
• Stereotyping: On BBQ, GRPO achieved the highest accuracy (0.3123 vs. 0.2823 for DPO).
• Knowledge Retention: Crucially, BiasGRPO increased its TruthfulQA score (0.3941 vs. 0.3843 for the base model), demonstrating that bias mitigation did not degrade factual knowledge or cause catastrophic forgetting.
• Training Stability: Analysis of training curves showed that while DPO plateaued early (limited generalization) and PPO exhibited jagged, unstable convergence (due to critic noise), BiasGRPO rose steadily with low variance. The standard deviation of rewards for BiasGRPO averaged 0.0668, less than half that of PPO (0.1434).

Ablation Studies

• Reward Model Robustness: When trained with a human-annotated alternative reward model (instead of the custom LLM-annotated one), GRPO still yielded significant improvements over the base model, confirming the algorithmic robustness of the group-relative mechanism.
• Group Size: Increasing the group size from $G=2$ to $G=8$ consistently improved performance across benchmarks, with $G=4$ serving as the standard. The underperformance of $G=2$ confirmed that the benefits stem from the group-relative normalization rather than online exploration alone.

Significance and Claims

The paper claims that BiasGRPO successfully addresses the specific limitations of existing bias mitigation strategies by leveraging the inherent fit between group-relative optimization and the subjective, high-variance nature of social bias.

Key contributions include:

1. Algorithmic Fit: Demonstrating that the foundational GRPO algorithm is superior to DPO and PPO for bias mitigation because it stabilizes training in noisy reward landscapes without sacrificing the exploration benefits of online learning.
2. Resource Availability: Releasing a diverse, multi-domain dataset and a highly efficient (0.1B parameter) custom bias reward model. This resource is designed to be "plug-and-play," allowing researchers to integrate robust bias mitigation into multi-objective RLHF pipelines without incurring significant compute overhead or conflicting with other alignment objectives.
3. Safety vs. Utility: Showing that effective bias mitigation can be achieved without knowledge degradation, as evidenced by the simultaneous improvement in bias metrics and TruthfulQA scores.

The authors conclude that the group-relative mechanism provides a stable foundation for bias mitigation, lowering the barrier to entry for implementing robust, multi-objective alignment strategies.