🤖 AI

Refining Over Resampling: Test-Time Self-Correction for LLM Reasoning

This paper proposes a verifier-free "breadth-depth" framework that enhances LLM reasoning by using test-time compute to iteratively refine diverse sampled rollouts through self-critique and self-correction, achieving superior performance across multiple mathematical benchmarks compared to traditional sampling and verifier-based selection methods.

Original authors: Ahsan Bilal, Muhammad Ahmed Mohsin, Muhammad Umer, Lena Trigg, Ali Subhan, Muhammad Ali, Dean F. Hougen

Published 2026-08-07

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ahsan Bilal, Muhammad Ahmed Mohsin, Muhammad Umer, Lena Trigg, Ali Subhan, Muhammad Ali, Dean F. Hougen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a really tricky math puzzle, like a riddle that requires a long chain of logic to crack. You have a super-smart friend (a Large Language Model, or LLM) who can talk to you. Usually, when you ask this friend a question, they give you one answer immediately. But sometimes, they get stuck on the first step and keep making the same mistake all the way to the end.

To fix this, scientists have tried two main tricks. The first is "resampling": asking your friend to try the puzzle ten times and picking the answer that shows up most often. It's like asking ten different people to solve the same riddle and seeing who agrees. The second trick is using a "verifier": a separate, super-strict teacher who grades every answer to see which one is best. But here's the catch: asking ten people often just gets you ten slightly different versions of the same wrong idea, and relying on a teacher means you have to trust that the teacher isn't confused or biased.

This paper, titled "Refining Over Resampling," suggests a third way. Instead of just asking for more guesses or hiring a teacher, the authors propose teaching the AI to be its own editor. They let the AI generate a few different starting ideas, and then, for each idea, they let the AI pause, critique its own work, find its own mistakes, and fix them before giving the final answer. It's like giving your friend a second chance to think, not by asking them to start over, but by saying, "Hey, wait, you made a small error in step three; let's fix that and keep going."

The Problem: The "Echo Chamber" and the "Hallucination Floor"

The researchers noticed something weird happening when they asked AI models to solve math problems by just generating more and more answers. They called this "diversity saturation." Imagine you ask a group of friends to guess the number of jellybeans in a jar. If you ask ten people, you might get ten different guesses. But if you ask a hundred people, you might find that ninety of them are just guessing numbers that are very close to each other because they all saw the same "obvious" (but wrong) clue. The AI was doing the same thing: it kept generating variations of the same few wrong paths, wasting its brainpower on repetition instead of finding new solutions.

On top of that, there was the "hallucination floor." Even if the AI knew the right answer deep down, it might accidentally trip over a small arithmetic error or a logic slip while writing it out. Since every single guess was made independently, every single guess was equally likely to trip over that same kind of mistake. Asking for more guesses didn't help because it didn't fix the trip; it just gave you more trips.

The Solution: Breadth and Depth

The authors came up with a strategy they call "Breadth-Depth Refinement." Think of it like exploring a maze.

Breadth is about casting a wide net. Instead of just walking one path, the AI generates several different starting routes (called "rollouts"). This ensures they aren't all stuck in the same "echo chamber" of wrong ideas.

Depth is the real magic. Once they have these different routes, they don't just pick the best one immediately. Instead, they take each route and send it through a loop of self-improvement.

The Generator: The AI writes down the next part of the solution.
The Critic: The AI stops and acts like a strict editor, reading what it just wrote and saying, "Wait, that math doesn't add up," or "That logic doesn't make sense."
The Corrector: The AI listens to the critic and rewrites that specific part to fix the error.

They do this loop a few times (called "depth") for every single route. Finally, they take all the now-polished, self-corrected answers and let them vote. The answer that wins the vote is the final one.

What They Found

The team tested this idea on some very hard math competitions, like the AIME (American Invitational Mathematics Examination) and the MATH500 dataset. They compared their method against the old ways: just guessing once (Greedy), guessing eight times and voting (Majority Voting), or guessing eight times and letting a "teacher" AI pick the winner (Verifier-based Best-of-N).

The results were quite promising. Their method consistently beat the others. For example, when they used a smaller, less powerful AI model called Qwen2.5-1.5B, the old "teacher" method got about 29.6% of the MATH500 questions right. But with their new "self-editing" method, the same model jumped to 58.0% accuracy. That's nearly double! On the AMC (American Mathematics Competitions) test, the same small model went from 25.0% to 32.5%.

Even with bigger, smarter models, their method still improved the scores, though the gains were a bit smaller because the smart models were already doing a decent job on their own.

Why It Works (and Why It's Different)

The key insight here is that the AI doesn't need an outside teacher to tell it what's wrong. It can learn to spot its own mistakes if you give it the right structure. The paper suggests that the "critic" step is crucial; when they removed the part where the AI explicitly critiques its own work, the scores dropped. It turns out that just asking the AI to "try again" isn't enough; it needs to be told what to fix.

Also, this method is "training-free." They didn't have to teach the AI anything new or show it thousands of examples of how to correct itself. They just used the AI's existing brainpower in a smarter way during the test.

The Trade-off

There is a cost, though. This method takes more computer power and time. Because the AI has to generate, critique, and rewrite multiple times for each problem, it uses more "compute" (the fancy term for the computer's brainpower) than just guessing once. However, the authors calculated that for the amount of extra work it does, the improvement in accuracy is worth it, especially for smaller models that struggle to get the right answer on the first try.

In the end, the paper suggests that when we want AI to be smarter, we shouldn't just ask it to try harder or ask for more guesses. We should give it the time and structure to think, check its work, and fix its own mistakes before it hands us the final answer. It's a shift from "more is better" to "better is better."

Technical Summary: Refining Over Resampling

Problem Statement

The paper addresses the limitations of current test-time scaling strategies for Large Language Models (LLMs) in mathematical reasoning. While allocating additional inference compute via wider sampling (e.g., Best-of-N) or verifier-based selection improves performance, the authors identify two critical failure modes:

Diversity Saturation: Increasing the sample budget ( $N$ ) often yields diminishing returns. Instead of discovering new reasoning paths, additional samples tend to cluster around a few dominant semantic directions, revisiting existing answer patterns rather than expanding the solution space.
Per-Trace Hallucination & Verifier Dependence:
- Hallucination: Each independent rollout is subject to the same generation noise. If a model's "intended" answer is correct but perturbed by generation errors, simple resampling does not fix the underlying trajectory; it merely repeats the error.
- Verifier Calibration: Verifier-based selection (using Process Reward Models or Outcome Reward Models) relies heavily on the calibration of an external reward model. Calibration errors can directly degrade final answer selection, and these methods often require additional training or learned stopping policies.

The authors argue that test-time compute is more effectively utilized not just to sample more, but to refine sampled trajectories before aggregation.

Methodology: Breadth–Depth Refinement

The proposed framework is a training-free, verifier-free approach that utilizes test-time compute along two axes: Breadth (diverse initial attempts) and Depth (iterative self-correction).

Core Components

The method uses a single base model $\pi_\theta$ conditioned on role-specific prompts to act as three distinct agents:

Generator ( $\pi^G_\theta$ ): Produces reasoning continuations.
Critic ( $\pi^C_\theta$ ): Identifies logical, arithmetic, or structural errors.
Corrector ( $\pi^R_\theta$ ): Revises the reasoning trace based on the critique.

The Algorithm

Initialization (Breadth): $N$ independent reasoning rollouts are sampled from the generator at a fixed temperature $\tau > 0$ . This ensures diverse starting points to counteract diversity saturation.
Iterative Refinement (Depth): Each of the $N$ $N$ rollouts undergoes $D$ $D$ rounds of a three-stage cycle:
- Reasoning Continuation: The generator extends or rewrites the previous trace.
- Self-Critique: The critic analyzes the trace and produces a natural language critique. If no errors are found, it explicitly confirms the trace.
- Self-Correction: The corrector revises the trace conditioned on both the reasoning and the critique.
Aggregation: After $D$ refinement steps, the final corrected traces are extracted into candidate answers. The final prediction is determined by majority voting (plurality) over the $N$ refined answers.

Crucially, this process requires no external verifier, no reward model, and no additional training.

Key Contributions

Identification of Limitations: The authors formally characterize "diversity saturation" (where larger $N$ fails to increase unique reasoning directions) and "per-trace hallucination" (where width-only scaling cannot fix systematic generation errors).
Breadth–Depth Framework: They propose a structured refinement method that preserves diverse initial attempts (Breadth) while applying iterative self-critique and self-correction (Depth) to repair local errors before aggregation.
Empirical Validation: The method is evaluated across five mathematical benchmarks (AIME24, AIME25, AMC, OlympiadBench, MATH500) and four open-weight models (Qwen2.5-7B/1.5B, Ministral-8B, LLaMA-3.1-8B).

Experimental Results

The method consistently outperforms greedy decoding, majority voting (Maj@8), verifier-based Best-of-N (RM@8), beam search, and lookahead decoding.

Performance Gains:
- Qwen2.5-1.5B: On MATH500, accuracy improved from 29.6% (RM@8 baseline) to 58.0%. On AMC, it rose from 25.0% to 32.5%.
- Qwen2.5-Math-7B: Showed consistent gains over the strongest baselines on AIME25, AMC, OlympiadBench, and MATH500.
- Ministral-8B & LLaMA-3.1-8B: Demonstrated clear improvements, particularly on AMC and MATH500.
Ablation Studies:
- Self-Critique Necessity: Removing the explicit self-critique stage (w/o) resulted in lower performance compared to the full pipeline (w/), confirming that the critique provides useful intermediate feedback rather than just increasing token count.
- Compute Efficiency: While the method incurs higher inference costs (TFLOPs) due to multiple refinement passes, the compute-normalized gain ( $\eta$ ) is positive. The gains are most pronounced for smaller models (e.g., 1.5B), suggesting refinement unlocks latent reasoning abilities not reliably expressed in single-shot generations.
Refinement Dynamics: Analysis shows that refinement successfully recovers from wrong-to-correct states while keeping correct-to-wrong regressions limited. Diversity remains high across refinement depths, indicating the method does not collapse all rollouts into a single answer.

Significance and Claims

The paper claims that test-time compute is more effective when used to refine sampled trajectories rather than solely to sample more candidates or rely on verifier-guided selection.

Robustness: By aggregating refined rollouts via majority voting, the method is robust to noisy individual corrections. It does not depend on the calibration of an external reward model.
Model Agnosticism: The approach is applicable to any open-weight model without requiring additional training, reinforcement learning, or learned stopping policies.
Paradigm Shift: The work suggests a shift from "searching wider" to "refining deeper," arguing that useful reasoning evidence comes from allowing each attempt to repair unstable intermediate steps before the final aggregation.

The authors acknowledge that the method has a higher inference cost than greedy decoding or standard sampling, but argue that the significant accuracy gains, particularly for smaller models, justify the trade-off in scenarios where model parameters cannot be updated.