RewardBench 2: Advancing Reward Model Evaluation

1. Problem Statement

Reward Models (RMs) are critical components in the post-training of Large Language Models (LLMs), used for Reinforcement Learning from Human Feedback (RLHF), data filtering, and inference-time scaling (e.g., Best-of-N sampling). However, the ecosystem for evaluating RMs is nascent and misaligned with downstream performance.

• The Gap: Existing benchmarks (e.g., RewardBench, RM-Bench) often rely on prompts repurposed from downstream evaluations, leading to potential data contamination. Furthermore, high scores on these benchmarks do not always correlate with improved performance in downstream tasks like RLHF training.
• The Challenge: Simpler direct alignment algorithms are often reported to outperform complex RM-based RLHF, suggesting current evaluation metrics may not capture the nuances required for effective RM selection. There is a need for a benchmark that uses unseen human prompts, tests multi-skill capabilities, and correlates strongly with downstream utility.

2. Methodology: REWARDBENCH 2 Construction

The authors introduce REWARDBENCH 2, a new benchmark designed to be more challenging and predictive than its predecessors.

• Data Source: Unlike previous benchmarks that reuse prompts from downstream evaluations, REWARDBENCH 2 sources unseen human prompts primarily from the WildChat pipeline (real-world user interactions), ensuring independence from downstream test sets.
• Format: The benchmark uses a Best-of-4 format. For each prompt, there is one chosen (correct) completion and three rejected (incorrect/distractor) completions. This lowers the random baseline to 25% (vs. 50% in 2-choice benchmarks), providing more headroom for evaluation.
• Six Domains: The benchmark covers six distinct domains, three of which are improvements on existing datasets and three are new:
  1. Factuality: Detecting hallucinations and factual errors (using dual LLM-as-a-judge verification).
  2. Precise Instruction Following (IF): Testing strict adherence to constraints (e.g., "no letter 'u'"), verified by programmatic functions.
  3. Math: Open-ended math problems from middle school to college level, graded via majority voting and manual verification.
  4. Safety: Testing refusal of harmful requests vs. compliance, based on the CoCoNot taxonomy.
  5. Focus: Detecting on-topic vs. off-topic responses to general queries.
  6. Ties (New): A novel domain testing RM calibration when multiple answers are equally valid (e.g., "Name a color of the rainbow"). The RM must prefer correct answers over incorrect ones without arbitrarily preferring one correct answer over another.
• Scoring: Primary metric is accuracy (selecting the correct response). The "Ties" domain uses a weighted score measuring both correctness and the margin between correct and incorrect answers.

3. Key Contributions

1. A Challenging, Unseen Benchmark: REWARDBENCH 2 forces models to generalize to new data. Leading models that score highly on RewardBench drop by an average of 20 points on REWARDBENCH 2.
2. Controlled Training Experiments: The authors trained 120+ new RMs to analyze the impact of:
   • Base Model Lineage: The stage of the base model (SFT vs. DPO vs. Instruct) significantly impacts RM performance.
   • Training Data: Combining Tulu and Skywork preference data yields better results than using either alone.
   • Epochs: Contrary to the "one-epoch" best practice, training for two epochs often improves performance without harming downstream utility.
3. Downstream Correlation Analysis: The paper rigorously quantifies how benchmark scores translate to two key use cases:
   • Inference-time Scaling (Best-of-N): Strong correlation found.
   • RLHF Training (PPO): Identified a critical "on-policy" vs. "off-policy" factor.

4. Key Results

Performance on the Benchmark

• Difficulty: Top existing models (e.g., Skywork-Reward-V2) score around 84% on average, but struggle significantly in specific domains:
  • Precise IF: ~40-50% accuracy.
  • Math: ~60-70% accuracy.
  • Factuality: ~75-85% accuracy.
• Model Lineage: RMs initialized from Llama 3.1 Instruct or Qwen 2.5 Instruct base models generally outperform those initialized from raw SFT models, indicating that post-training capabilities carry over to the RM.

Downstream Correlation Findings

• Best-of-N (BoN) Sampling: REWARDBENCH 2 scores show a high Pearson correlation (0.87) with downstream BoN performance. This suggests the benchmark is an excellent proxy for selecting RMs for inference-time scaling.
• RLHF (PPO) Training:
  • The "Lineage" Trap: The paper identifies a crucial finding: A high-scoring RM on the benchmark does not guarantee good RLHF performance if the RM and the Policy Model are from different lineages.
  • On-Policy vs. Off-Policy: When an RM is trained on a base model different from the policy model (e.g., using a Llama-based RM to train a Tulu-based policy), downstream performance drops significantly, even if the RM scored highly on REWARDBENCH 2.
  • Saturation: For "on-policy" RMs (same lineage), performance saturates quickly; even moderate-performing RMs achieve similar downstream results to top-performing ones.

5. Significance and Implications

• Paradigm Shift in Evaluation: The paper argues that accuracy-based benchmarks are necessary but insufficient. The context of the training setup (specifically the alignment between the RM's base model and the policy model) is a dominant factor in RLHF success.
• Practical Guidance:
  • For Best-of-N sampling, users can rely on REWARDBENCH 2 scores to pick the best off-the-shelf RM.
  • For RLHF, users should not simply pick the highest-scoring RM. Instead, they should replicate the training recipe (base model + data mix) of the top-performing RM within their own specific policy lineage.
• Community Impact: By releasing the benchmark data (ODC-By) and code (Apache 2.0), the authors provide a standardized, harder, and more realistic testbed to drive the development of more robust reward models that generalize better to real-world scenarios.

In summary, REWARDBENCH 2 advances the field by moving beyond simple accuracy on static datasets to a multi-skill, unseen-prompt evaluation that reveals the critical importance of model lineage alignment in downstream RLHF applications.