Technical Summary: Self-Verifying Refinement (SVR)

Problem Statement

Scaling test-time computation is a central strategy for improving language model reasoning, yet current approaches face two primary limitations:

1. Inefficient Resource Allocation: Fixed-budget methods (e.g., uniform multi-turn refinement or fixed-$k$ sampling) waste computation on easy inputs that are already solved and may fail to allocate sufficient resources to difficult inputs.
2. Reliance on External Signals: Existing adaptive or self-correction methods often rely on external verifiers, execution environments, or oracle feedback (ground-truth correctness) to decide when to stop refining. These signals are often costly, unavailable at deployment, or introduce a disconnect between training and inference information boundaries.

The core challenge is enabling a model to learn an internal control signal to decide whether to retain a current answer or allocate further refinement turns, using only signals generated by the policy itself, without access to ground-truth correctness during inference.

Methodology: Self-Verifying Refinement (SVR)

The authors propose Self-Verifying Refinement (SVR), an oracle-free, multi-turn reinforcement learning framework that learns to use self-verification as a compute-control policy.

1. Adaptive Refinement Formulation

SVR treats test-time computation as a sequential allocation problem. At each turn $t$, the policy $\pi_\theta$ generates a structured output $o_t = (r_t, a_t, v_t, c_t)$, consisting of:

• $r_t$: Reasoning trace.
• $a_t$: Task answer.
• $v_t$: Discrete verdict ($Correct$, $Incorrect$, or $Unsure$).
• $c_t$: Confidence score (probability estimate that $a_t$ is correct).

Inference Mechanism:
The inference-time controller operates on an oracle-free information boundary. It stops and returns the current answer $a_t$ if and only if:

1. The generation is not truncated ($q_t = 0$).
2. The verdict is $Correct$($v_t = C$).
3. The confidence exceeds a deployment threshold $\gamma$ ($c_t \ge \gamma$).

If these conditions are not met and the budget remains, the model constructs the next prompt using the original problem, the retained draft, and the policy's own self-verification state, without ever accessing ground-truth labels.

2. Training: Joint Verdict–Confidence Reinforcement Learning

SVR is trained using Group Relative Policy Optimization (GRPO) on fixed-horizon trajectories ($T_{tr}$). Ground-truth correctness is used only to construct training rewards and is never exposed to the policy during the refinement process.

The trajectory-level return $R_{SVR}$ averages three per-turn signals:

• Solution Reward ($r_{solve}$): Rewards absolute correctness and cross-turn progress (e.g., correcting errors or preserving correct answers) while penalizing regression and truncation.
• Self-Verification Reward ($r_{verify}$): A composite objective designed to align the verdict-confidence pair with empirical correctness:
  • Calibration: A Brier-style objective to align confidence $c_t$ with correctness $y_t$.
  • Overconfidence Penalty: Penalizes high-confidence $Correct$ verdicts on incorrect answers.
  • Error Detection: Rewards explicit $Incorrect$ verdicts on wrong answers.
  • Stop-Readiness: Encourages high confidence on correct answers to facilitate early stopping.
• Format Reward ($r_{fmt}$): Ensures valid task-answer formatting and parseable self-check structures.

Key Training Design:

• Fixed-Horizon Collection: During training, all trajectories run for the full $T_{tr}$ turns regardless of intermediate verdicts. This prevents erroneous early stopping from cutting off the policy's exposure to necessary refinement states.
• Optimization Scope: The policy-gradient loss is applied only to the final completion tokens of the trajectory. Intermediate turns contribute to the trajectory return (supervising the self-verification signal) but are not treated as independent optimization samples.

3. Inference

At inference, the trained policy is deployed with an adaptive scheduler. The model generates turns sequentially, evaluating the stopping condition after each turn. The training horizon $T_{tr}$ and deployment budget $T_{max}$ are independent; $T_{tr}$ ensures sufficient supervision coverage, while $T_{max}$ bounds the worst-case cost.

Key Contributions

1. Oracle-Free Closed-Loop Framework: SVR introduces a method where a single policy learns both to refine answers and to generate the structured self-verification signal (verdict + confidence) required for inference-time control, eliminating the need for external verifiers or oracle feedback during deployment.
2. Joint Verdict–Confidence RL Objective: The authors develop a multi-turn objective that aggregates solution quality, calibration-aware self-verification, and stop-readiness into a single trajectory return. This trains the policy to produce actionable stopping signals without directly optimizing the realized inference cost during training.
3. Empirical Validation: The framework demonstrates that learned self-verification can serve as an effective internal control signal for adaptive compute allocation, achieving a superior accuracy-compute trade-off compared to fixed-budget and oracle-guided baselines.

Experimental Results

The method was evaluated using Qwen3.5-2B across seven mathematical reasoning benchmarks (Countdown, GSM8K, MATH500, AIME26, AMC23, OlympiadBench, MinervaMath).

• Performance: SVR achieved a macro-average accuracy of 0.563 on the All-7 benchmarks.
• Efficiency: It required an average of only 2.99 inference turns per example.
• Comparisons:
  • Outperformed the strongest non-oracle multi-turn baseline (Murphy) by 7.5 percentage points in accuracy.
  • Surpassed a fixed-budget oracle-guided reference (which uses external correctness feedback) by 4.4 percentage points while using significantly fewer tokens.
  • Matched the accuracy of 10-sample GRPO majority voting while consuming approximately half the token cost.
• Ablation Studies:
  • Removing the self-verification reward block ($R_{verify}$) reduced accuracy by ~3.5 points, confirming the necessity of the control signal.
  • Removing the "stop-readiness" component increased the average turns to 3.83 and reduced accuracy by 6.9 points, highlighting the importance of training the model to recognize when it is ready to stop.
  • Single-turn SVR (no iterative refinement) dropped accuracy significantly, proving that iterative refinement is essential.
  • Using only verdicts or only confidence resulted in suboptimal performance compared to the joint signal.

Significance and Claims

The paper claims that SVR demonstrates learned self-verification can effectively replace external oracle signals for controlling test-time compute. By treating structured self-verification not merely as a post-hoc description of uncertainty but as a policy-level signal, SVR enables models to:

• Retain correct intermediate answers that might otherwise be overwritten by further refinement.
• Allocate computation adaptively based on instance difficulty, avoiding the waste of fixed-budget approaches.
• Operate in a "oracle-free" manner at inference, making the approach deployable in settings where external verifiers are unavailable.

The authors note that while SVR significantly improves the accuracy-compute frontier, reducing Premature Stop Errors (PSE) on difficult problems remains an open challenge, as the model still occasionally terminates refinement on incorrect answers with high confidence. The work suggests that the integration of self-verification into the reasoning policy itself is a viable path toward efficient, adaptive reasoning without external supervision.