Technical Summary: Off-the-Shelf LLMs as Process Scorers

Problem Statement

Current strategies for improving mathematical reasoning in large language models (LLMs) face a trade-off between cost and effectiveness. Using a massive model for every inference query is computationally expensive. A common alternative is to sample multiple solutions from a smaller, cheaper model and select the best one post-hoc (after generation is complete) using methods like Majority Voting or Best-of-N. However, the paper argues that post-hoc selection is fundamentally limited: if the small model diverges into an incorrect reasoning path early in generation, no scorer can correct the trajectory once the error has propagated.

Conversely, Process Reward Model (PRM) guided search addresses this timing issue by scoring candidate reasoning steps during generation. While effective, PRMs require a reward model trained on step-level labels (e.g., Qwen2.5-Math-PRM-72B), which is a significant training overhead and resource cost.

The core problem this paper addresses is: Can we achieve the benefits of generation-time guidance (steering the model before errors propagate) without training a dedicated reward model?

Methodology: Chunk-Level Guided Generation

The authors propose Chunk-Level Guided Generation, a training-free framework that utilizes an off-the-shelf large language model (LLM) solely as a scorer, not a generator.

The Framework

1. Generation: A small model ($\pi_s$) generates the solution. At each step, instead of generating a single token or a variable-length reasoning step, the small model samples $k$ candidate continuations (chunks) of a fixed length $L$.
2. Scoring: A larger, pre-trained LLM ($\pi_l$) scores these $k$ candidates. Crucially, the large model does not generate text; it only computes likelihoods (log-probabilities) for the candidate chunks provided by the small model.
3. Selection: The highest-scoring chunk is appended to the context, and the process repeats until an end-of-sequence token is reached.

Scoring Rules

The paper introduces two selection rules based on the large model's likelihoods:

• Likelihood-Guided Selection (LGS): Selects the chunk with the highest length-normalized log-probability under the large model:
  $$S_{LGS}(c_i) = \ell_{\pi_l}(c_i | h)$$
• Contrastive-Guided Selection (CGS): Selects the chunk where the large model's preference diverges most from the small model's. It subtracts the small model's log-probability from the large model's:
  $$S_{CGS}(c_i) = \ell_{\pi_l}(c_i | h) - \ell_{\pi_s}(c_i | h)$$
  This rule favors continuations that the large model finds plausible but the small model does not, effectively leveraging the large model's "extra" knowledge.

Key Design Choice: Fixed-Length Chunks

A critical contribution of the methodology is the use of fixed-length chunks rather than naturally delimited reasoning steps. The authors demonstrate that large-model log-probabilities exhibit a systematic length bias: even after length normalization, the large model tends to assign higher probabilities to longer reasoning steps. By fixing the length $L$ for all candidates at a decision point, the scores become directly comparable, removing this confound.

Key Contributions

1. Training-Free Process Scoring: The paper demonstrates that an off-the-shelf LLM, used exclusively as a likelihood scorer, is an effective substitute for a trained PRM in guiding generation-time search.
2. Resolution of Length Bias: The authors identify and resolve a systematic length bias in large-model likelihoods, showing that fixed-length chunks are necessary for reliable scoring, whereas variable-length step scoring is unreliable.
3. Contrastive Selection: The introduction of CGS, which leverages the difference in preferences between the large and small models, provides consistent gains over simple likelihood maximization.
4. Efficiency: The method produces substantially shorter reasoning traces than PRM-guided search while achieving comparable or superior accuracy.

Experimental Results

The framework was evaluated on five mathematical reasoning benchmarks: GSM8K, MATH, Minerva Math, AMC23, and AIME24. The experiments utilized three model pairs:

• Qwen2.5-1.5B guided by Qwen2.5-32B
• Llama-3.2-1B guided by Llama-3.1-70B
• Qwen2.5-7B guided by Qwen2.5-72B

Performance vs. Baselines

• vs. Post-Hoc Selection: Chunk-Level Guided Generation (specifically CGS) substantially outperforms post-hoc methods like Majority Voting, Self-Certainty, and Best-of-N. For example, on GSM8K with the Llama pair ($k=32$), CGS achieved 83.9% accuracy compared to 56.0% for Majority Voting (a 27.9 percentage point gain).
• vs. PRM Guided Search: Under matched guidance budgets (comparing the number of interventions), CGS matches or outperforms PRM guided search using Qwen2.5-Math-PRM-72B on most benchmarks without any reward-model training.
  • On Minerva Math (Qwen pair, $k=32$), CGS achieved 39.0% vs. 32.4% for PRM.
  • On GSM8K (Llama pair, $k=32$), CGS achieved 83.9% vs. 77.3% for PRM.
• Scaling: When the generator was scaled to Qwen2.5-7B (guided by 72B), CGS reached 81.8% on MATH and 63.6% on Minerva Math, surpassing Majority Voting by 4–6 percentage points and coming within 1.5 percentage points of the PRM-72B guided search average.

Reasoning Trace Length

Contrary to the assumption that better performance requires longer reasoning, the paper finds that CGS produces substantially shorter reasoning traces than PRM guided search. On GSM8K, PRM responses contained up to 80% more tokens than CGS responses. This suggests the large model's likelihood guidance steers the small model toward more concise, direct reasoning paths.

Significance and Claims

The paper claims that Chunk-Level Guided Generation offers a practical, training-free alternative to PRM-guided search. Its significance lies in:

1. Accessibility: It removes the barrier of training a step-level reward model, allowing practitioners to use existing, off-the-shelf large models to improve smaller models.
2. Timing: It successfully intervenes during generation to prevent error propagation, a capability previously thought to require specialized training.
3. Efficiency: It achieves high accuracy with shorter reasoning traces, reducing inference costs compared to PRM methods.

The authors remain modest regarding generalization, noting that experiments are currently limited to mathematical reasoning and within-family model pairs (e.g., Qwen-to-Qwen, Llama-to-Llama). They do not claim the method generalizes to coding or commonsense reasoning without further investigation.