Process Supervision via Verbal Critique Improves Reasoning in Large Language Models

1. Problem Statement

Current methods for scaling inference-time compute in Large Language Models (LLMs) rely on three established axes:

1. Chain Depth: Increasing tokens per trajectory (e.g., Chain-of-Thought).
2. Sample Breadth: Aggregating parallel trajectories (e.g., Self-Consistency, Tree-of-Thoughts).
3. Learned Step-Scorers: Training Process Reward Models (PRMs) on labeled step correctness.

The authors identify a fourth, overlooked axis: the granularity of external verbal supervision. Existing verbal feedback methods (like Reflexion) typically provide outcome-level critique (feedback on the final answer only). This conflates credit across all reasoning steps, making it difficult to isolate specific errors. The paper posits that moving from outcome-level to step-level verbal supervision without requiring training or gradient updates is a distinct and powerful scaling mechanism.

2. Methodology: Verbal Process Supervision (VPS)

The paper introduces Verbal Process Supervision (VPS), a training-free, inference-time framework that instantiates the step-level end of the verbal supervision axis.

• Core Components:
  • Actor ($\pi_\theta$): A base model that generates reasoning trajectories.
  • Supervisor ($C$): A stronger model (or the same model with higher reasoning effort) that evaluates the trajectory.
• The Loop (Generate–Critique–Refine):
  1. The actor generates a trajectory $\tau_r$.
  2. The supervisor produces a structured, natural-language critique $c_r$ at specific steps (not just the end).
  3. The actor regenerates a refined trajectory $\tau_{r+1}$ conditioned on the previous trajectory and the critique: $\tau_{r+1} \sim \pi_\theta(\cdot | x, \tau_r, c_r)$.
  4. This loop repeats for $R$ rounds or until a stopping criterion is met.
• Key Distinctions from Prior Work:
  • Vs. Reflexion: Reflexion provides a single end-of-episode signal. VPS provides dense, step-indexed feedback, preserving correct sub-trajectories and localizing refinement to specific failure points.
  • Vs. PRMs: VPS replaces trained scalar reward models with verbal critiques from a stronger LLM, eliminating the need for annotation data and training costs.
  • Training-Free: No gradient updates or parameter tuning are required; the actor's weights remain fixed.

3. Key Contributions

1. Identification of a New Scaling Axis: The paper formalizes "critique granularity" as a distinct axis of inference-time scaling, orthogonal to depth, breadth, and learned scoring.
2. Formalization of VPS: Defines VPS as a language-space actor-critic update operator with fixed-point convergence characteristics.
3. Empirical Validation: Demonstrates that at matched inference compute, step-level critique significantly outperforms outcome-level critique and sample aggregation.
4. Characterization of Regimes: Identifies a "weak-actor rescue" regime and establishes a strong correlation between supervisor capability gaps and performance gains.

4. Experimental Results

The framework was evaluated on three benchmarks: GPQA Diamond (scientific reasoning), AIME 2025 (mathematics), and LiveCodeBench V6 (competitive programming).

A. Performance vs. State-of-the-Art (SOTA)

• GPQA Diamond: The GPT-5.4 (High) | GPT-5.4 (Low) configuration achieved 94.9% accuracy at $R=4$, surpassing the reported SOTA of 94.1% and the standalone baseline of 92.8%. This was achieved purely through same-family verbal self-refinement.
• AIME 2025: VPS unlocked dramatic capability amplification in "weak-actor" scenarios. Actors with standalone scores of 11.7–26.7% were lifted to 63.3–90.0%, yielding absolute gains of up to +63.3 points.
• LiveCodeBench V6: VPS improved scores from 38.5% to 50.0% for the GPT-5.4 Mini pair.

B. Baseline Comparisons (Matched Compute)

To ensure gains were due to the mechanism rather than just extra compute, VPS was compared against:

• Self-Consistency @ 5 (SC@5): Sampling 5 trajectories and voting.
• Reflexion: Outcome-level verbal feedback.

Results:

• Vs. Reflexion: VPS outperformed Reflexion by +8.5 to +12.1 points across all benchmarks. This isolates granularity as the operative variable, proving step-level feedback is superior to outcome-level feedback.
• Vs. SC@5: VPS outperformed SC@5 by +5.0 pp on GPQA and +8.3 pp on LiveCodeBench. On AIME, VPS narrowly exceeded SC@5 (+1.1 pp), suggesting VPS is not reducible to simple sample aggregation.

C. Key Observations & Limitations

• Non-Monotonic Scaling: Performance does not always improve linearly with round count ($R$). Optimal $R$ varies by model pair (e.g., GPT-5.4 peaked at $R=4$, while GLM-5.1 on AIME peaked at $R=2$ then regressed due to over-correction).
• Supervisor-Actor Gap: Gains are strongly predicted by the capability gap between supervisor and actor (Pearson $r = +0.90$). VPS is most effective when the supervisor can identify errors the actor cannot.
• Domain Boundary: VPS showed diminishing returns on code synthesis (LiveCodeBench) for open-weight models. The authors argue this is a principled boundary: verbal critique is effective when errors are linguistically representable (e.g., math, logic) but less effective when errors are only detectable via runtime execution (code).

5. Significance

This paper establishes critique granularity as a fundamental lever for improving LLM reasoning without training costs.

• Efficiency: It offers a path to SOTA performance using only inference-time compute and stronger models, avoiding the high cost of training PRMs.
• Mechanism: It proves that the structure of feedback (step-level vs. outcome-level) is more critical than the mere presence of verbal feedback.
• Future Direction: The findings motivate hybrid verbal-executable supervision, where runtime test-case feedback is integrated into the verbal critique signal to overcome the limitations in code synthesis tasks.

In summary, VPS demonstrates that a stronger model providing granular, step-by-step verbal guidance can effectively "rescue" weaker models, creating a scalable, training-free pathway to enhanced reasoning capabilities.