Reward Models Can Improve Themselves: Reward-Guided Adversarial Failure Mode Discovery for Robust Reward Modeling

1. Problem Statement

Reward Modeling (RM) is a critical component in aligning Large Language Models (LLMs) with human preferences, used in tasks like fine-tuning (RLHF/DPO), response ranking, and test-time alignment. However, current reward models suffer from significant vulnerabilities:

• Distributional Shifts & Adversarial Perturbations: They often fail when encountering inputs slightly outside their training distribution (e.g., capitalization changes, verbosity, misspellings, or repetition).
• Reward Hacking: Models can be "hacked" to assign high rewards to harmful or low-quality responses that exploit superficial patterns (spurious correlations) rather than true semantic quality.
• Limitations of Existing Detection: Current methods for identifying these failure modes rely on counterfactual generation using external LLMs (like GPT-4) guided by prior knowledge of specific attributes (e.g., "length," "tone"). This approach is impractical because:
  1. It requires expensive external API calls.
  2. It assumes prior knowledge of which attributes cause failures, which is often unavailable or model-specific.
  3. It fails to generalize across different model architectures and training distributions.

Core Research Question: Can reward model failure modes be identified tractably without prior preference knowledge, and can the reward model use these discovered failures to improve its own robustness?

2. Methodology: The REFORM Framework

The authors propose REFORM (Reward-Guided Adversarial Failure Mode Discovery), a self-improving framework consisting of three main stages:

A. Reward-Guided Controlled Decoding (Failure Mode Discovery)

Instead of using external prompts, REFORM uses the reward model itself to guide an aligned policy to generate "failure modes."

• Goal: Generate responses that belong to a known preference class (e.g., a "preferred" response) but receive an incorrect score (e.g., a low reward) from the reward model.
• Mechanism:
  1. Base Policy: Start with a policy ($\pi_{\mathcal{D}}$) aligned to the preference distribution (for generating false negatives) or a misaligned policy (for generating false positives).
  2. Candidate Selection: At each decoding step, select the top-$k$ tokens from the base policy.
  3. Adversarial Steering: Instead of maximizing the reward (standard alignment), the algorithm minimizes the reward for preferred responses (or maximizes for non-preferred) while maintaining linguistic coherence.
  4. Objective Function:
     $$\arg \min_{y^{(i)} \in v^k_i} \left[ r_\phi(x, [y_{<i}, y^{(i)}]) + \alpha \cdot \log \pi_{\mathcal{D}}(y^{(i)} | x, y_{<i}) \right]$$
     Where $r_\phi$ is the reward model, and $\alpha$ is a hyperparameter balancing reward minimization with the likelihood of the aligned policy (ensuring fluency).
• Result: This produces class-consistent but reward-inconsistent examples (e.g., a harmless response that the model scores as harmful, or vice versa).

B. Data Augmentation & Selection

• Influential Point Selection: The method identifies the top 5% of training samples with the lowest Bradley-Terry loss (indicating they are the most influential on the current reward model's decision boundary).
• Failure Variant Generation: For these influential pairs, the controlled decoding process generates "failure variants" (perturbed versions of the preferred/non-preferred responses).
• Filtering: Variants are filtered to ensure they are true failure cases (e.g., a preferred response variant that now scores lower than the rejected response).

C. Self-Improving Fine-tuning

• The original training dataset is augmented with these newly discovered failure variants.
• The reward model is retrained (or fine-tuned) on this failure-aware dataset.
• This process forces the model to learn that the specific perturbations (e.g., specific capitalization patterns or verbosity) do not change the true preference, thereby "patching" the misaligned behavior.

3. Key Contributions

1. Preference-Distribution-Agnostic Discovery: A novel method to discover failure modes without relying on external LLMs or prior knowledge of specific failure attributes. It leverages the reward model's own internal logic to find its weaknesses.
2. Self-Improving Framework (REFORM): Demonstrates that a reward model can iteratively improve its own robustness by using its own failures as training data.
3. Data Efficiency: The method achieves significant robustness improvements by augmenting only the top 5% of influential training samples, avoiding the need for massive-scale data collection or manual relabeling.
4. Preservation of Utility: Proves that increasing robustness against adversarial perturbations does not degrade the model's performance on standard, unperturbed data or downstream alignment tasks.

4. Experimental Results

The framework was evaluated on two safety-critical datasets: Anthropic Helpful-Harmless (HH) and PKU Beavertails, using Mistral 7B and Qwen 2.5 14B models.

• RQ1: Failure Mode Detection:
  • REFORM outperformed baselines (random perturbation and attribute-based prompting) in generating class-appropriate failure modes.
  • It successfully identified "false negatives" (preferred responses getting low scores) where attribute-based methods failed, highlighting the importance of model-dependent discovery.
• RQ2: Robustness:
  • REFORM-trained models showed significantly higher robustness against four types of perturbations: Verbosity, Capitalization, Repetition, and Misspellings.
  • The "win rate drop" (a measure of vulnerability) was substantially lower for REFORM compared to standard reward models and other robust training methods (like RRM).
• RQ3: Downstream Alignment Quality:
  • Reward Accuracy: The REFORM reward model maintained high accuracy on unperturbed test data.
  • Alignment Performance: When used to train policies via BoN (Best-of-N), PPO, and DPO, the resulting models preserved readability, diversity, and utility.
  • Spurious Correlation Removal: The method effectively reduced the model's tendency to rely on spurious correlations (e.g., penalizing harmless responses simply because they were verbose or contained specific capitalized words).

5. Significance and Implications

• Autonomous Robustness: REFORM shifts the paradigm from static reward modeling to dynamic, self-correcting systems. It suggests that reward models can "learn" their own vulnerabilities and fix them without human intervention.
• Safety Criticality: By making reward models robust to adversarial perturbations (like capitalizing harmful words to bypass filters), this work directly addresses a major safety gap in deploying LLMs in real-world scenarios.
• Cost-Effectiveness: The method is computationally efficient compared to generating massive counterfactual datasets with external LLMs, making it scalable for large-scale alignment pipelines.
• Generalizability: The approach is model-agnostic regarding the type of failure, as it does not require pre-defining what the failure looks like, making it applicable to a wide range of reward modeling architectures.

In conclusion, the paper demonstrates that reward models are not static evaluators but can be evolved into robust, self-improving systems by systematically identifying and correcting their own adversarial failure modes.