💻 computer science

Rethinking Reward Models for Multi-Domain Test-Time Scaling

This paper challenges the conventional wisdom that fine-grained process reward models are superior by presenting a unified evaluation across 14 domains, which reveals that generative outcome reward models (gORM) are the most robust and consistent performers, while stepwise generative process models suffer from error compounding due to label noise.

Original authors: Dong Bok Lee, Seanie Lee, Sangwoo Park, Minki Kang, Jinheon Baek, Dongki Kim, Dominik Wagner, Jiongdao Jin, Heejun Lee, Tobias Bocklet, Jinyu Wang, Jingjing Fu, Sung Ju Hwang, Jiang Bian, Lei Song

Published 2026-07-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dong Bok Lee, Seanie Lee, Sangwoo Park, Minki Kang, Jinheon Baek, Dongki Kim, Dominik Wagner, Jiongdao Jin, Heejun Lee, Tobias Bocklet, Jinyu Wang, Jingjing Fu, Sung Ju Hwang, Jiang Bian, Lei Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to teach a super-smart robot how to solve tricky puzzles. You don't just want the robot to get the right answer; you want to know how it got there. Did it take a shortcut? Did it make a mistake, realize it, and fix itself? Or did it just guess and get lucky?

To help the robot learn, we use a "Reward Model"—think of it as a strict teacher grading the robot's work. For a long time, the smartest teachers in the math world were Process Reward Models (PRMs). These teachers graded every single step of the robot's thinking process. If the robot made a tiny mistake in step 3, the teacher would stop and say, "Game over, this is wrong!" The idea was that grading every step was like having a magnifying glass; it seemed like the most detailed, helpful way to learn.

But here's the twist: This paper suggests that for most real-world puzzles (like law, medicine, or general knowledge), that magnifying glass might actually be blinding the teacher.

The Big Experiment: 14 Different Worlds

The researchers didn't just look at math problems. They tested four different types of "teachers" across 14 diverse domains (including law, biology, history, and psychology). They wanted to see which teacher was the best at helping the robot pick the best solution from a bunch of guesses.

The four teachers were:

The Final-Answer Grader (dORM): Only looks at the final result. "Did you get the right answer? Yes/No."
The Step-by-Step Grader (dPRM): Checks every step. "Step 1 is good, Step 2 is bad, STOP."
The Chatbot Final-Answer Grader (gORM): A smart AI that writes a little explanation and gives a final verdict.
The Chatbot Step-by-Step Grader (gPRM): A smart AI that writes an explanation for every single step and grades them one by one.

The Surprise Results

In the world of math, the Step-by-Step teachers (PRMs) usually win. But when the researchers moved to the other 13 domains, the results flipped completely:

The "Step-by-Step" Chatbot (gPRM) was the worst. It struggled to keep up.
The "Step-by-Step" Human-style teacher (dPRM) was just okay. It performed about the same as the simple Final-Answer grader.
The "Chatbot Final-Answer" teacher (gORM) was the champion. It was the most robust, giving consistent improvements across every single domain tested.

Why Did the Step-by-Step Teachers Fail?

The paper offers two main reasons, and they are pretty clever.

1. The "Aha!" Moment Problem
Imagine a robot solving a puzzle. It makes a mistake in step 2, but then it realizes, "Oh no, I messed up!" and fixes it in step 3, eventually getting the right answer. This is called an "Aha!" moment.

The PRM Problem: The Step-by-Step teachers are trained to think that if one step is wrong, the whole thing is wrong. They are like a referee who blows the whistle the second a player trips, even if the player gets up and scores the winning goal. The paper shows that in these multi-domain tests, these teachers often miss the "Aha!" recoveries because they stop grading too early.
The Length Issue: The longer the robot's thinking chain gets, the more likely the Step-by-Step teacher is to make a mistake in its own grading. The paper suggests that as the chain of reasoning gets longer, the error in the Step-by-Step teacher's score grows, making it less reliable.

2. The "Noisy Label" Trap
In math, we have perfect human teachers who can grade every step perfectly. But in law or medicine, it's too expensive to hire humans to grade every single step of a robot's thinking. So, we use other AIs to grade the steps automatically.

The Problem: These automatic graders make mistakes (noise). The paper found that the Step-by-Step teachers are very sensitive to these mistakes. If the automatic grader says "Step 3 is wrong" when it's actually right, the Step-by-Step teacher gets confused and fails.
The Winner: The Final-Answer teachers (especially the Chatbot version, gORM) are much tougher. They can ignore the little noise in the middle and focus on whether the final result makes sense.

The "Consensus Filter" Glitch

There's one more reason the Step-by-Step Chatbot (gPRM) failed. To train it, researchers used a method called "consensus filtering." They asked the AI to grade its own steps, but if the AI's grading didn't match the "ground truth" (the answer key), they threw that data away.

The Result: This process accidentally threw away all the long, complicated, "Aha!"-style thinking chains. It left the teacher with only short, simple examples to learn from. When the teacher tried to grade a long, complex puzzle in the real world, it was totally unprepared. The paper shows that the training data for this teacher was very different from the test data, like teaching a student only with 5th-grade math and then testing them on 12th-grade calculus.

The Takeaway

If you are building a system to help robots think through complex, real-world problems (like legal cases or medical advice), don't obsess over grading every single step.

The paper suggests that a Generative Outcome Reward Model (gORM)—a smart AI that explains its reasoning and gives a final "Yes/No" verdict on the whole solution—is the most reliable tool. It's robust, handles long chains of thought better, and isn't thrown off by noisy data.

While the Step-by-Step teachers (PRMs) are great for math where we have perfect labels and short problems, the paper argues that for the messy, long, and complex real world, the "Big Picture" teacher (gORM) is the one you want in your corner.

Technical Summary: Rethinking Reward Models for Multi-Domain Test-Time Scaling

Problem Statement

Test-time scaling (TTS) enables Large Language Models (LLMs) to improve reliability by generating multiple reasoning chains (CoTs) and selecting the best candidate using external verifiers. While Outcome Reward Models (ORMs) and Process Reward Models (PRMs) have shown success in math-adjacent domains, their efficacy across broader, high-stakes domains (e.g., legal, medical) remains underexplored. Furthermore, the comparative performance of discriminative versus generative verifier architectures in these multi-domain settings is unclear. Existing literature often assumes that fine-grained process supervision (PRMs) is universally superior, but this assumption relies heavily on math-specific datasets with high-quality manual annotations.

Methodology

The authors present the first unified, controlled evaluation of four reward model variants across 14 diverse domains (including MMLU-Pro, GPQA-Diamond, MedQA, and LEXam). The four variants are:

dORM (Discriminative ORM): A classifier trained on outcome labels (correct/incorrect final answer) only.
dPRM (Discriminative PRM): A classifier trained on step-level process labels, aggregating scores (typically via minimum) to determine trajectory correctness.
gORM (Generative ORM): An LLM fine-tuned to generate a verification CoT and a binary verdict ("Yes"/"No"), trained via consensus filtering where the generated verdict aligns with the ground-truth outcome.
gPRM (Generative PRM): An LLM fine-tuned to generate step-wise verification rationales and process verdicts, also trained via consensus filtering.

Experimental Setup:

Domains: Math (PRM800K, ProcessBench) and Multi-domain/Specialized (MMLU-Pro, GPQA-Diamond, MedQA, LEXam).
Training Data: Math labels were human-annotated; multi-domain process labels were automatically generated by LLMs (introducing potential noise).
Evaluation: The models were used as verifiers in a Best-of-N (BoN) selection framework, where $N$ CoTs are generated, and the one with the highest reward score is selected.
Analysis: The authors conducted theoretical analysis on error bounds relative to CoT length and empirical analysis on label noise sensitivity and distribution shifts induced by consensus filtering.

Key Contributions

Unified Multi-Domain Evaluation: The paper provides the first controlled comparison of dORM, dPRM, gORM, and gPRM across 14 domains, moving beyond the math-centric focus of prior work.
Counter-Intuitive Findings: The study challenges the conventional wisdom that PRMs are always superior. It finds that in multi-domain settings:
- dORM performs on par with dPRM.
- gPRM is not competitive and often underperforms other variants.
- gORM is the most robust, yielding significant and consistent gains across every tested domain.
Theoretical and Empirical Analysis of Failure Modes:
- CoT Length Risk: The authors prove (Theorems 4.2 & 4.3) that the log-error lower bound for PRMs grows linearly with CoT length ( $T$ ), whereas ORM error bounds remain independent of $T$ . Empirically, PRM performance degrades as CoTs lengthen, particularly failing to recognize "aha" moments (trajectories that recover from early errors).
- Label Noise and Distribution Shift: In multi-domain settings, reliance on LLM-based auto-labeling introduces noise. The authors demonstrate that gPRM suffers from a severe distribution shift in CoT length due to consensus filtering, which prunes long, complex reasoning chains, leading to a mismatch between training and test distributions.

Results

Math Domain: Results align with prior work: dPRM outperforms dORM, and generative models (gORM/gPRM) generally outperform discriminative ones.
Multi-Domain & Specialized Domains:
- dORM vs. dPRM: dORM achieves performance comparable to dPRM, suggesting that outcome-level supervision is sufficient when process labels are noisy or domains are diverse.
- gORM Superiority: gORM consistently outperforms all other variants, including gPRM.
- gPRM Failure: gPRM fails to compete, largely due to the compounding of errors in step-wise scoring and the inability to handle long, self-correcting reasoning trajectories.
Robustness: gORM demonstrates superior robustness to label noise and varying CoT lengths compared to PRM variants.

Significance and Claims

The paper argues that the assumption "fine-grained supervision is always better" does not hold in multi-domain settings. The authors claim that generative outcome verification (gORM) is the most reliable strategy for deploying TTS in diverse, high-stakes domains.

Key implications highlighted by the authors include:

Re-evaluating PRMs: The stepwise aggregation mechanism in PRMs compounds errors as reasoning length increases, making them unsuitable for domains where long, complex reasoning (including self-correction) is common.
Practical Guidelines: The authors provide specific recommendations:
- Use gORM for mixed/shifting domains, long CoTs, or when reliability is paramount (e.g., medical/legal).
- Use dORM/dPRM only when CoTs are short, labels are clean, and strict latency/compute budgets exist.
- Avoid gPRM in multi-domain settings due to its sensitivity to length distribution shifts and label noise.

The work concludes that for multi-domain deployment, generative outcome verification offers a more robust and consistent path forward than process-based supervision, challenging the prevailing trend toward granular process labeling in complex reasoning tasks.