VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Models

1. Problem Statement

The emergence of Large Reasoning Models (LRMs) like OpenAI's o1 and DeepSeek-R1 has shifted the reinforcement learning (RL) paradigm from standard preference-based optimization to reference-based reward systems. In these systems, model outputs are evaluated against ground-truth reference answers (e.g., correct math solutions or logical deductions) rather than relative preferences between two responses.

However, existing reward benchmarks (e.g., RewardBench, RM-Bench) focus almost exclusively on pairwise preference comparisons (ranking Response A vs. Response B). They fail to evaluate the critical capability of absolute verification: determining whether a single response is correct or incorrect based on a reference answer. This gap hinders the systematic evaluation and improvement of the reward systems used to train state-of-the-art reasoning models.

2. Methodology

The authors introduce VerifyBench and its challenging variant VerifyBench-Hard, designed specifically to evaluate reference-based reward systems.

A. Benchmark Construction

The construction pipeline involves three main stages:

1. Query Curation:
   • VerifyBench: Aggregated from 41 distinct open-source datasets covering General, Logical, and Mathematical reasoning.
   • VerifyBench-Hard: Derived from the same pool but filtered to include only "contentious" cases where top-performing LLMs exhibit high disagreement in their judgments.
2. Completion Generation:
   • Multiple open-source and proprietary LLMs (22 models for VerifyBench, 18 for VerifyBench-Hard) generated single-shot completions for the curated questions.
3. Annotation & Filtering:
   • Answer Type Labeling: Questions were classified into four canonical types: Numerical Values, Algebraic Expressions, Multiple-Choice, and Free-form Strings.
   • Human Annotation: At least two human annotators verified the correctness of each completion against the ground truth. Disagreements were resolved by meta-annotators.
   • Balancing:
     • VerifyBench: Constructed with a balanced distribution (1,000 unique questions, each paired with exactly one correct and one incorrect completion) to ensure fair evaluation across answer types.
     • VerifyBench-Hard: Derived via natural sampling of difficult cases, resulting in a skew toward incorrect completions (291 correct vs. 709 wrong) to reflect real-world difficulty.

B. Evaluation Framework

The paper defines a new evaluation metric for reference-based reward models ($R_\phi$). Unlike pairwise metrics, the accuracy is measured by the model's ability to predict a binary correctness label ($y$) for a single response ($r$) given a query ($q$) and ground truth ($gt$):
$$\text{Accuracy} = \frac{1}{|D|} \sum I [E(R_\phi(q, gt, r)) = y]$$
where $E(\cdot)$ maps continuous reward scores to discrete correctness predictions.

3. Key Contributions

1. VerifyBench & VerifyBench-Hard: The first dedicated benchmarks for evaluating reference-based reward systems, moving beyond pairwise preference to absolute correctness verification.
2. High-Quality Curation: A meticulously curated dataset of 2,000 balanced samples (VerifyBench) and 1,000 difficult samples (VerifyBench-Hard) with rigorous human verification.
3. Comprehensive Empirical Analysis:
   • Evaluation of diverse baselines: Rule-based functions (e.g., math-verify), LLM-as-a-Judge, and specialized model-based verifiers.
   • Ablation Study: Demonstrated that providing reference answers in the prompt is critical, with performance dropping 5–18% when references are omitted.
   • Correlation Analysis: Proved that verifiers performing well on VerifyBench lead to better downstream RL training outcomes.

4. Key Results

• Performance Gap: While state-of-the-art LLMs (e.g., GPT-4o, Qwen3-32B) achieve high accuracy (~93–95%) on the standard VerifyBench, performance drops significantly on VerifyBench-Hard (best accuracy ~72.4%). This indicates a 20% performance gap, highlighting substantial room for improvement in handling ambiguous or complex reasoning.
• Model Size Sensitivity: Smaller models (<3B parameters) struggle significantly, with accuracies as low as 25–44% on VerifyBench-Hard, whereas larger models (>70B) maintain robust performance.
• Error Categories: The analysis identified specific failure modes:
  • Numeric Values: Errors with complex numbers or multi-value answers where ordering is irrelevant.
  • Expressions: Failure to recognize mathematical equivalence (e.g., unsimplified expressions).
  • Strings: Difficulty in verifying semantic consistency rather than exact string matching.
• Impact on RL Training: Experiments using different verifiers to train a base model (Qwen2.5-3B) via GRPO showed a strong correlation: verifiers with higher VerifyBench scores produced RL-trained models with significantly better performance on downstream math tasks (MATH500, AMC23). Conversely, poor verifiers degraded model performance.

5. Significance

• Fills a Critical Gap: Provides the first standardized framework for evaluating the specific reward mechanisms used in modern reasoning model training (RLVR), which were previously unassessed by existing benchmarks.
• Guides Future Development: By identifying that current verifiers struggle with "hard" cases and small-scale models, the paper directs future research toward improving verification robustness and efficiency.
• Practical Utility: The correlation analysis confirms that VerifyBench is not just a theoretical metric but a practical tool for selecting and training reward models that directly enhance the reasoning capabilities of LLMs.

In conclusion, VerifyBench establishes a new standard for the evaluation of reference-based reward systems, offering a principled foundation for advancing the reliability and accuracy of reinforcement learning in large reasoning models.