Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization

1. Problem Statement

While Reinforcement Learning from Human Feedback (RLHF) has successfully aligned Large Language Models (LLMs) with general human values (e.g., correctness, helpfulness, safety), a critical gap remains in pluralistic alignment: the ability to account for diverse, individual user preferences.

• Limitations of Current Benchmarks: Existing reward model (RM) benchmarks (e.g., RewardBench) focus on general quality. They often construct "rejected" responses by injecting errors or using lower-quality models, conflating general quality with personal preference.
• The "Proxy Gap": There is a lack of benchmarks that demonstrate a strong correlation between a reward model's static accuracy and its ability to improve downstream policy performance (e.g., via Best-of-N sampling or PPO).
• The Core Challenge: Evaluating whether a reward model can distinguish between two high-quality responses where the only differentiator is adherence to a specific user's unique rubrics, style, and constraints.

2. Methodology: Personalized RewardBench

The authors introduce Personalized RewardBench, a novel benchmark designed to rigorously assess reward models on personalized alignment.

A. Dataset Construction

• Source: The benchmark is built upon the LaMP-QA dataset, covering three high-variance domains: Arts & Entertainment, Lifestyle & Personal Development, and Society & Culture.
• Data Structure: Each instance is a tuple $(q, u, y_c, y_r)$, consisting of a query ($q$), a user profile ($u$), a chosen response ($y_c$), and a rejected response ($y_r$).
• Preference Pair Generation (Key Innovation):
  • Unlike prior work that generates "bad" answers, this benchmark generates two high-quality responses.
  • Chosen Response ($y_c$): Generated by an LLM (Gemini-3-Flash) strictly adhering to user-specific rubric aspects ($R_u$) derived from the user's history.
  • Rejected Response ($y_r$): Generated by the same LLM explicitly violating those specific rubric aspects ($\neg R_u$), while maintaining high general fluency and correctness.
  • Result: The discriminative factor is isolated strictly to personal preference alignment, not general quality.

B. Human Evaluation

The authors conducted rigorous human evaluations across four dimensions:

1. Factuality & Correctness
2. Relevance & Instruction Following
3. Helpfulness & Harmlessness
4. Personal Rubrics Alignment

Findings: Both chosen and rejected responses scored similarly high (4.3–4.99/5) on general quality metrics. However, the chosen responses scored significantly higher on Personal Rubrics (4.8–4.9) compared to rejected responses (1.4–1.5), confirming the benchmark successfully isolates personalization as the differentiator.

C. User Profile Integration Strategy

Since standard RMs struggle to process raw user profiles (leading to train-test misalignment), the authors propose a Planner-based approach:

• A dedicated Planner module ($P$) infers structured rubric aspects ($\tilde{R}_u$) from the user's history ($u$) and current query ($q$).
• The Reward Model then evaluates responses using these inferred rubrics: $s = RM(q, \tilde{R}_u, y)$.
• This approach outperforms both the baseline (no profile) and naive direct injection of raw profiles.

3. Key Contributions

1. Personalized RewardBench: A new benchmark that explicitly incorporates user profiles and personal rubrics, shifting evaluation from general quality to user-specific alignment.
2. Methodological Rigor: A construction method where rejected answers are high-quality but fail specific personal constraints, ensuring the reward signal captures personalization rather than error detection.
3. Downstream Correlation Validation: The first comprehensive demonstration that a reward model's performance on a static benchmark correlates significantly with downstream policy performance (BoN and PPO).
4. Identification of the Personalization Gap: Empirical evidence showing that even state-of-the-art models struggle with personalized alignment.

4. Experimental Results

A. Benchmark Performance

• State-of-the-Art Struggle: Even the best-performing model, Gemini-3-Flash, achieved only 75.94% accuracy. Most other models (e.g., Skywork, InternLM, R3) scored between 60% and 73%.
• Scaling Laws Decoupling: Increasing model parameters does not guarantee better personalization. For instance, the 20B parameter internlm2 model performed worse than its 7B counterpart, suggesting personalization requires specialized training objectives rather than just scale.

B. Downstream Correlation (The "Proxy Gap" Analysis)

The authors evaluated the correlation between benchmark rankings and downstream policy performance using Best-of-N (BoN) and Proximal Policy Optimization (PPO).

• Metrics: NDCG, Rank Biased Overlap (RBO), Weighted $\tau$, and Spearman's $\rho$.
• Findings:
  • Personalized RewardBench showed superior correlation with downstream performance (e.g., NDCG of 0.9180 for BoN and 0.9265 for PPO).
  • Baselines (Chatbot Arena-Personalized, PRISM) showed weak or even negative correlations (e.g., Chatbot Arena had a negative Weighted $\tau$ of -0.0736 for BoN), indicating they fail to predict which reward models actually improve policy outputs.

5. Significance and Conclusion

• New Standard for Evaluation: Personalized RewardBench establishes a robust proxy for evaluating reward models in real-world, personalized applications. It proves that high accuracy on general benchmarks does not translate to effective personalization.
• Future Research Direction: The significant gap between current model performance (76%) and the theoretical oracle limit (98%) highlights a massive opportunity for developing reward models specifically trained on subjective preference distributions.
• Practical Impact: By validating that this benchmark predicts downstream success, the paper provides a reliable tool for researchers to select and tune reward models for personalized LLM deployment, moving beyond "one-size-fits-all" alignment.