💬 NLP

Scaling Evaluation-time Compute with Reasoning Models as Evaluators

This paper demonstrates that employing reasoning models as evaluators with increased test-time compute—specifically through step-by-step process evaluation—significantly improves evaluation accuracy and can enhance an LM's problem-solving capabilities through reranking, mirroring the benefits of scaling compute during generation.

Original authors: Seungone Kim, Ian Wu, Jinu Lee, Xiang Yue, Seongyun Lee, Mingyeong Moon, Carolin Lawrence, Kiril Gashteovski, Julia Hockenmaier, Graham Neubig, Sean Welleck

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Seungone Kim, Ian Wu, Jinu Lee, Xiang Yue, Seongyun Lee, Mingyeong Moon, Carolin Lawrence, Kiril Gashteovski, Julia Hockenmaier, Graham Neubig, Sean Welleck

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are taking a difficult math test. Usually, to get a better score, you might try to think harder or write out more steps to solve the problem. This paper asks a fascinating question: What if, instead of just thinking harder about the answer, you thought harder about grading the answer?

Here is the core idea of the paper, broken down with simple analogies:

1. The Old Way: The "Fast Grader"

Traditionally, when we use AI to check if another AI's answer is right, we use a "Direct Evaluator." Think of this like a fast-food cashier. You hand them a receipt (the answer), and they instantly say, "That looks good" or "That looks bad." They don't stop to think about why it's good or bad; they just give a quick score based on patterns they've seen before.

2. The New Idea: The "Reasoning Grader"

The researchers tried something different. They used a special type of AI called a Reasoning Model to do the grading. Think of this like a tenured professor who refuses to just glance at the answer.

Instead of giving a quick score, this "professor" forces itself to write out a long, detailed essay (a "Chain of Thought") explaining its reasoning. It might say:

"Wait, let me check step 3 again..."
"Hmm, this logic seems shaky."
"Let me backtrack and see if there's a better way."
"Okay, I've verified the whole thing, and I'm confident."

The paper calls this "Scaling Evaluation-Time Compute." In plain English: Spending more time and brainpower to grade the test, rather than spending more time trying to solve the test.

3. The Two Ways They Graded

The researchers tested two ways for this "professor" to grade:

Outcome Evaluation: Looking at the final answer and asking, "Is the result correct?" (Like checking the final score on a test).
Process Evaluation: Looking at every single step of the solution and asking, "Is this specific step logical?" (Like checking the work shown on the test paper).

The Big Discovery: The "professor" (Reasoning Evaluator) got much better at grading the more it was forced to "think" and write out its reasoning steps. Just like a student gets better at solving math problems by thinking longer, the grader gets better at finding mistakes by thinking longer.

4. The "Best-of-N" Game: Quality vs. Quantity

To test if this better grading actually helps, they played a game called Best-of-N.

The Setup: Imagine a generator AI creates 64 different answers to a hard problem.
The Old Strategy: Use a "Fast Grader" to quickly check all 64 answers and pick the best one.
The New Strategy: Use the "Reasoning Grader" (the professor) to deeply analyze only 8 answers, but analyze them very carefully.

The Result: The "Reasoning Grader" picking the best of just 8 answers performed better than the "Fast Grader" picking the best of 64 answers.

The Analogy: It's like hiring one expert detective to investigate 8 clues very thoroughly, rather than hiring 100 interns to glance at 64 clues. The deep investigation found the truth more often than the wide, shallow search.

5. Why Does This Work?

The paper found that the "Reasoning Grader" is especially good at spotting hidden errors.

Sometimes an AI gets the right final answer but used a wrong method to get there (like guessing the right number on a math test).
The "Fast Grader" might say, "Great job!" because the answer is right.
The "Reasoning Grader" looks at the steps, sees the mistake, and says, "Wait, the answer is right, but the logic is flawed," and rejects it.

Summary

The paper proves that spending more computing power to evaluate an answer is just as effective as spending more power to generate the answer.

By forcing the AI grader to "think out loud" and check every step carefully, we can get better results with fewer attempts. It's a shift from "trying harder to get the right answer" to "thinking harder to find the right answer."

Technical Summary: Scaling Evaluation-Time Compute with Reasoning Models as Evaluators

Problem Statement

The paper addresses a gap in the current paradigm of Large Language Model (LM) development. While it is well-established that scaling "test-time compute" (e.g., generating long chains-of-thought, CoT, or sampling multiple candidates) significantly improves the problem-solving capabilities of generators, it remains unclear whether this scaling principle applies to evaluators. Current evaluators often rely on direct scoring (Outcome Reward Models, ORMs) or short CoT reasoning (Process Reward Models, PRMs) without fully leveraging the computational power available at inference time. The authors ask: Can an LM's evaluation capability be improved by scaling test-time compute, and can this improved evaluation subsequently enhance the generator's performance?

Methodology

The authors propose a framework for Evaluation-Time Scaling using Reasoning Evaluators. Unlike traditional direct evaluators that predict scores without reasoning or fine-tuned generative evaluators with short CoTs, reasoning evaluators are off-the-shelf reasoning models (trained via RL or distillation to produce long CoTs) prompted to act as judges.

The methodology involves three core components:

Reasoning Outcome Evaluators: The model generates a long CoT to analyze the entire response before predicting a correctness score (0 or 1). The score is derived from the logits of the prediction tokens.
Reasoning Process Evaluators: The model evaluates the response step-by-step.
- Multi-step Evaluation: Instead of generating a single CoT for the whole response, the evaluator assesses each reasoning step individually, generating a distinct CoT and judgment for each.
- Model-based Splitting: To handle unstructured outputs (e.g., code), the authors use an LM to insert [SPLIT] markers between steps, rather than relying on heuristic newline splitting.
- Aggregation: Step-wise scores are aggregated using a mean_logit function rather than a simple minimum, which proved more effective in experiments.
Combined Evaluation: The final score is a weighted interpolation of the outcome score and the aggregated process score ( $s_{final} = \alpha \cdot s_{outcome} + (1-\alpha) \cdot s_{process}$ ), with $\alpha=0.5$ used to balance both signals.

The authors compare these reasoning evaluators against:

Direct Evaluators: Standard PRMs and ORMs that predict scores directly without CoT.
Single-step Reasoning Evaluators: Models that generate one CoT to judge all steps simultaneously (an ablation of their proposed method).

Key Contributions

Demonstration of Evaluation-Time Scaling: The paper establishes that forcing evaluators to generate more reasoning tokens (scaling evaluation-time compute) leads to monotonic improvements in evaluation performance.
Superiority of Multi-step Process Evaluation: The authors show that evaluating each step individually (multi-step) is significantly more effective than single-step evaluation or self-consistency (generating multiple CoTs for the same task) under fixed compute budgets.
Zero-Shot Performance: Reasoning evaluators, which are not explicitly trained as evaluators, outperform state-of-the-art, heavily fine-tuned PRMs (e.g., a 32B reasoning evaluator outperforms a 72B PRM on ProcessBench) solely through test-time scaling.
Generator Improvement via Evaluation: The paper demonstrates that allocating more compute to evaluation (using Best-of-8 with reasoning evaluators) yields better generator performance than allocating that same compute to generating more candidates (Best-of-64 with direct evaluators). This suggests that a stronger, more compute-intensive evaluator mitigates "reward model over-optimization."

Experimental Results

Process Evaluation (ProcessBench)

Metric: F1 score in identifying the first erroneous step in a solution.
Findings:
- Reasoning models (e.g., DeepSeek-R1-Distill-Qwen-32B) significantly outperform instruction-tuned models of similar or larger sizes.
- Multi-step vs. Single-step: Multi-step process evaluation yields substantial gains over single-step methods. For instance, the 32B reasoning model achieved an F1 of 78.6 with multi-step evaluation, surpassing the 72B Qwen2.5-Math-PRM (78.3), which was the previous state-of-the-art.
- Self-Consistency: While self-consistency helps, multi-step evaluation provides complementary benefits. Combining both yields the highest scores (e.g., 82.8 for the 32B model).

Problem-Solving (Best-of-N Sampling)

Setting: Generators (Eurus-2, Llama-3.1-70B, Qwen2.5-7B) generate $N$ responses. Evaluators rank them.
Comparison: Reasoning evaluators (Best-of-8) vs. Direct evaluators (Best-of-64) under a fixed total compute budget.
Findings:
- Reasoning evaluators using Best-of-8 outperform direct evaluators using Best-of-64 by 4.30% to 6.63% across benchmarks (AIME, MATH, LeetCode, etc.).
- Code Performance: Reasoning evaluators show particularly strong gains on coding tasks (LeetCode), where direct PRMs often suffer from over-optimization and poor domain transfer.
- Combined Scores: Integrating process and outcome scores further boosts performance, acting as a tie-breaker for responses that outcome evaluators cannot distinguish.

Self-Evaluation

The paper also explores using a reasoning model to generate and evaluate its own outputs. The results indicate that reasoning models can effectively self-evaluate, recovering a significant portion of the gap between their Best-of-1 and oracle performance, often outperforming instruction-tuned generators evaluated by the same reasoning model.

Significance and Claims

The paper claims that evaluation-time scaling is a viable and powerful strategy that parallels test-time scaling for generators. The primary significance lies in the finding that spending more compute on evaluation can be more effective than spending it on generation for a fixed budget.

The authors argue that:

Reasoning Capability Transfers: The ability to reason (via long CoT) is not just for solving problems but is intrinsic to high-quality evaluation.
Robustness: Reasoning evaluators are less prone to reward model over-optimization, providing more robust signals for selecting the best candidate.
Generalization: This approach works across math and code domains and even extends to non-reasoning models when prompted to evaluate step-by-step.

The paper concludes that while current reasoning evaluators are not explicitly trained for evaluation, their native reasoning patterns (self-verification, backtracking) make them superior judges. Future work is suggested to explore training these models specifically as evaluators and extending the methodology to non-verifiable tasks like creative writing.