Mitigating Reward Hacking in RLHF via Advantage Sign Robustness

1. Problem Statement: Reward Hacking in RLHF

Reinforcement Learning from Human Feedback (RLHF) aligns Large Language Models (LLMs) with human intent using a Reward Model (RM) as a proxy for true human preferences. However, RMs are imperfect proxies. As the policy optimizes to maximize the proxy reward, it often exploits local inaccuracies in the reward landscape, a phenomenon known as reward hacking.

• The Core Issue: The policy increases the probability of responses that yield high proxy rewards but low true quality.
• The Mechanism: The paper posits that reward hacking is frequently caused by flipped advantage signs. If the RM incorrectly predicts the sign of the advantage (e.g., predicting a positive advantage for a bad response), the policy update reinforces the bad response instead of suppressing it.
• Limitations of Prior Work: Existing solutions are either ante-hoc (requiring multiple RMs or changes to the RM training pipeline, which is computationally expensive) or post-hoc methods that still require access to the original RM training data (e.g., AdvPO). There is a need for a lightweight, purely post-hoc method that operates solely during policy optimization without retraining the RM.

2. Methodology: Sign-Certified Policy Optimization (SignCert-PO)

The authors propose SignCert-PO, a method that down-weights policy gradient updates for completions where the advantage sign is fragile (i.e., easily flipped by small perturbations in the RM).

A. Theoretical Foundation: Certified Sign-Preservation Radius

The method draws inspiration from Randomized Smoothing (RS) used for adversarial robustness. Instead of perturbing input data, the authors model uncertainty by perturbing the RM parameters.

• Definition: For a given completion $j$, the certified sign-preservation radius ($\Delta_j$) is defined as the smallest perturbation to the RM parameters that would flip the sign of the advantage $A_j$.
• Interpretation: A large $\Delta_j$ implies the advantage sign is robust (reliable). A small $\Delta_j$ implies the sign is fragile and likely unreliable (a candidate for reward hacking).

B. Practical Approximation: Linear Head Perturbation

Calculating $\Delta_j$ for all RM parameters is computationally infeasible due to memory constraints (requiring per-completion full-parameter gradients).

• Simplification: The authors assume the RM consists of a frozen feature extractor $h_\psi(x, y)$ and a learnable linear head $w^T h + b$. They only consider perturbations to the linear head parameters ($w$).
• Closed-Form Solution: Under this assumption, the certified radius admits a closed-form solution:
  $$\Delta_j = \frac{|A_j|}{\|h_\psi(x, y^{(j)}) - \bar{h}\|_2}$$
  Where $\bar{h}$ is the mean feature vector of the group of completions.
  • Intuition: The radius is small if the advantage $|A_j|$ is small (weak signal) or if the feature vector $h_j$ is an outlier (large deviation from the mean), indicating the RM is extrapolating in an uncertain region.

C. The Algorithm: Re-weighted Policy Gradient

SignCert-PO modifies the standard Group Relative Policy Optimization (GRPO) gradient.

1. Adversarial Formulation: For each completion, an adversary chooses a perturbation $w'$ within a radius $\epsilon$ to maximize the "damage" (minimize positive advantages, maximize negative ones).
2. Worst-Case Advantage: The worst-case advantage $\tilde{A}_j$ is derived as:
   $$\tilde{A}_j = \rho^*_j \cdot A_j$$
   Where the re-weighting coefficient is:
   $$\rho^*_j = 1 - \frac{\epsilon}{\Delta_j}$$
   (If $\rho^*_j < 0$, the term is effectively suppressed or clipped).
3. Implementation:
   • The threshold $\epsilon$ is set adaptively based on the empirical distribution of the ratio $\|h_j - \bar{h}\|_2 / |A_j|$ within a batch (using a quantile $q_t$).
   • The policy gradient is updated using the re-weighted advantages: $\nabla J \propto \sum \rho^*_j A_j \nabla \log \pi$.

D. Comparison with Global Robustness

The paper contrasts SignCert-PO with Global Robust Optimization (applying a single worst-case perturbation to all completions).

• Global Approach: Results in a uniform penalty term that does not distinguish between reliable and unreliable completions, leading to "uniform pessimism" and suboptimal performance.
• SignCert-PO: Provides per-completion discrimination, selectively suppressing only the unreliable gradient directions while preserving updates for robust completions.

3. Key Contributions

1. Novel Formulation: Introduces the concept of certified sign-preservation radius in the RM parameter space to quantify the reliability of advantage signs per completion.
2. Lightweight Post-Hoc Method: Proposes SignCert-PO, which requires only the current RM head parameters and on-policy completions. It does not require:
   • Retraining the RM.
   • Access to the original RM training dataset.
   • Multiple RM ensembles.
3. Theoretical Derivation: Derives a closed-form solution for the certified radius under a linear head assumption, making it computationally efficient.
4. Empirical Validation: Demonstrates that the certified radius predicts robustness even under broader perturbation classes (e.g., full-model perturbation, input embedding noise).

4. Experimental Results

The method was evaluated on TL;DR Summarization and AlpacaFarm benchmarks using Pythia and Qwen2.5 models.

• Performance: SignCert-PO consistently achieved the highest gold-RM win rate across most settings, outperforming baselines like Dr.GRPO, Uncertainty Weighted Optimization (UWO), Behavior-Supported PO (BSPO), and AdvPO.
• Mitigating Reward Hacking:
  • KL-Reward Trade-off: While baselines (like Dr.GRPO) showed a collapse in true reward (gold reward) as KL divergence increased (indicating hacking), SignCert-PO maintained high true reward even at higher KL budgets.
  • RM Accuracy: SignCert-PO kept the policy in regions where the proxy RM remained accurate (~60% agreement with gold RM), whereas baselines drifted into regions where RM accuracy dropped to ~50%.
• Data Scarcity: The method provided the largest gains when the proxy RM was trained on limited preference data (e.g., 1 epoch), a regime where reward hacking is most severe.
• Efficiency: SignCert-PO added negligible computational overhead compared to standard GRPO, as it only requires calculating feature deviations and re-weighting coefficients, avoiding the heavy cost of RM ensembles or dataset access.

5. Significance

• Practicality: SignCert-PO solves the reward hacking problem without the heavy infrastructure costs of training multiple RMs or retaining large datasets, making it highly deployable in real-world RLHF pipelines.
• Safety & Alignment: By explicitly targeting the sign of the advantage (the direction of the update), it prevents the policy from being pulled toward "spurious" high-reward regions that are artifacts of the RM's limitations.
• Theoretical Insight: The work bridges the gap between adversarial robustness (certified radii) and RLHF, offering a new perspective that reliability is a per-sample property rather than a global model property.

In conclusion, SignCert-PO offers a mathematically grounded, efficient, and highly effective mechanism to stabilize RLHF training, ensuring that policy updates are driven by reliable signals rather than fragile proxy rewards.