Technical Summary: OPENREWARD

Problem Statement

Reward Models (RMs) are critical for aligning Large Language Models (LLMs) during both training and inference, serving as scalable proxies for human evaluation. However, existing RMs, whether trained as binary preference classifiers or utilizing the "LLM-as-a-Judge" paradigm, struggle significantly with open-ended, knowledge-intensive, long-form tasks. Evaluating such content (e.g., medical reports, travel itineraries, or deep research outputs) often requires grounding in external information that exceeds the model's internal parametric knowledge. Current approaches either rely on prompting commercial LLMs with search engines (lacking flexibility) or use simple tool interactions for short-form verification, leaving a gap in effectively evaluating complex, long-form responses in open-domain settings.

Methodology

The paper proposes OPENREWARD, a tool-augmented reward modeling framework designed to adaptively reason and call external tools to gather evidence before making a judgment.

1. Agentic Reward Framework

OPENREWARD operates as a sequential decision process. Given a query $q$ and two candidate responses ($x_1, x_2$):

• Tool Invocation: The model iteratively decides which tool to invoke (e.g., Wikipedia Search, arXiv Search, PubMed Search) based on the current context.
• Evidence Gathering: Selected tools execute to retrieve external results ($e_i$), which are appended to the context to guide subsequent reasoning.
• Final Judgment: After gathering sufficient evidence, the model selects the better response ($y$) and provides an explanation.

2. Training via Reinforcement Learning (RL)

To train the model to effectively utilize tools and make accurate judgments, the authors employ Group Relative Policy Optimization (GRPO).

• Training Signal: A composite reward function $R$ is designed to supervise both intermediate behavior and final outcomes:
  $$R = R_{EM} + \text{sign}(R_{EM}) \cdot \lambda \cdot R_{tool}$$
  • $R_{EM}$ (Outcome Accuracy): A binary reward (1 or 0) based on whether the final judgment matches the ground truth.
  • $R_{tool}$ (Tool Usage): A reward of +1 for correct tool selection and 0 for incorrect/irrelevant calls.
  • Crucial Design: The tool reward is gated by the final outcome accuracy ($\text{sign}(R_{EM})$). This prevents "reward hacking" where the model might learn to call tools frequently but still fail to answer correctly.
• Optimization: GRPO computes group-relative advantages by comparing trajectories within a group, encouraging the discovery of effective multi-step reasoning strategies through trial-and-error.

3. Controllable Data Synthesis (OPENRM-LIB)

Addressing the scarcity of reliable pairwise data for long-form tasks, the authors introduce OPENRM-LIB, a framework for unsupervised data synthesis:

• Target-aware Query Generation: A strong LLM generates queries from sampled documents (e.g., Wikipedia, arXiv).
• Positive/Negative Pair Construction:
  • Positive Response: Generated by an LLM with access to the reference document.
  • Negative Response: Generated by the same LLM without access to the reference document.
• This contrastive setup ensures a clear quality gap (grounded vs. hallucinated/incomplete) without manual annotation, yielding over 27,000 high-quality training pairs.

Key Contributions

1. OPENREWARD Framework: Introduction of a tool-augmented RM capable of autonomously retrieving external evidence to judge complex, long-form responses, addressing the limitations of static internal knowledge.
2. Controllable Data Synthesis: A novel strategy (OPENRM-LIB) to generate 27K+ high-quality pairwise training data in an unsupervised manner, overcoming data bottlenecks for long-form evaluation.
3. RLVR Training Strategy: A composite reward function using GRPO that jointly optimizes for tool selection accuracy and final judgment correctness, effectively preventing reward hacking.
4. Empirical Validation: Comprehensive evaluation demonstrating that OPENREWARD outperforms existing baselines (including LLM-as-a-Judge and other trained RMs) across multiple domains.

Experimental Results

The paper evaluates OPENREWARD on three newly collected in-domain datasets (Wikipedia, Scientific, Medical) and two out-of-domain benchmarks (RewardBench, PandaLM).

• In-Domain Performance: OPENREWARD (based on Qwen-2.5-7B) achieves an average accuracy of 91.33% on the new datasets, substantially outperforming strong baselines like GPT-4o (62.80%), Claude-Opus (63.23%), and other trained RMs (e.g., RM-R1 at 54.32%).
• Out-of-Domain Generalization: Despite being trained on only 27K examples (compared to 100K+ for some baselines), OPENREWARD achieves 78.54% average accuracy on RewardBench and PandaLM, outperforming models trained on significantly larger datasets. The authors attribute this to the model's ability to reason over retrieved evidence rather than memorizing training distributions.
• Ablation Studies:
  • Removing the tool-reward signal leads to "lazy searching" (relying on internal knowledge).
  • Decoupling tool rewards from final accuracy leads to "reward hacking" (over-searching without correct judgments).
  • The gated reward design is essential for balancing reasoning and tool usage.
• Downstream Utility:
  • Data Selection: Using OPENREWARD to filter training data for DPO yields higher selection accuracy (75.18%) compared to RM-R1 (68.87%).
  • Model Alignment: LLMs aligned using OPENREWARD-selected data show consistent performance gains (e.g., +4.57% on average for the 7B model) over those aligned with existing RMs.

Significance

The paper posits that tool-augmented reward modeling represents a significant step forward for reliable evaluation and post-training alignment of LLMs. By enabling RMs to act as agentic systems that verify claims against external evidence, OPENREWARD addresses the fundamental limitation of standard RMs in knowledge-intensive scenarios. The work demonstrates that integrating external tools with reinforcement learning not only improves the accuracy of the reward model itself but also enhances the quality of downstream LLM alignment, suggesting a scalable path toward more trustworthy AI systems.