Technical Summary: Verifiable Process Supervision for Language Models

Problem Statement

Training language models (LMs) to produce both correct answers and sound reasoning remains a significant challenge. Current reinforcement learning with verifiable rewards (RLVR) approaches typically optimize solely for final outcome correctness. While this often improves task accuracy, it creates a failure mode where intermediate reasoning becomes incomplete, inaccurate, or internally inconsistent. The paper identifies a critical tension: outcome-based rewards can lead models to produce better predictions while their exposed reasoning traces degrade in quality. This is particularly problematic when the reasoning trace itself is required for debugging, auditing, or downstream reuse. Existing process supervision methods often rely on learned reward models or LLM-based judges, which introduce computational costs, calibration difficulties, and potential noise.

Methodology: Verifiable Process Supervision (VPS)

The authors propose Verifiable Process Supervision (VPS), a post-training framework designed for verifiable domains that jointly optimizes prediction accuracy and reasoning quality. The framework operates through three core components:

1. Structured Reasoning Prior via SFT:
   The process begins with Supervised Fine-Tuning (SFT) on synthetically constructed traces that follow a consistent, domain-specific schema. This induces a structured reasoning format, biasing the model to produce traces where intermediate claims appear in predictable positions. This structure enables the downstream verifier to extract claims using simple syntactic pattern matching rather than expensive semantic parsing or LLM judges.

2. Deterministic Verification and Reward Computation:
   During Reinforcement Learning (RL), a verifier extracts individual claims from the structured trace and evaluates them against ground-truth signals available in the domain (e.g., chess engine outputs, rule-based checks). This produces a set of subtask rewards ($r_k$). These are aggregated into a reasoning reward ($R_{reason}$) using the formula:
   $$R_{reason}(x, \tau) = \sum_{k=1}^{K} w_k \cdot r_k$$
   The total reward combines format adherence, prediction accuracy, and the reasoning reward:
   $$R(x, \tau, \hat{y}) = R_{form}(\hat{y}) + R_{acc}(\hat{y}, y^*) + \lambda \cdot R_{reason}(x, \tau)$$

3. Adaptive Reward Weighting:
   To address the heterogeneous difficulty of reasoning subtasks, the authors introduce an adaptive weighting mechanism. They track the model's performance on each subtask using an Exponential Moving Average (EMA) of batch rewards. Weights ($w_k$) are dynamically adjusted based on the "headroom" ($1 - \mu_k$) of each subtask, prioritizing components with the largest remaining errors. This creates an implicit curriculum that directs learning toward the most difficult subskills.

Experimental Setup

The framework is evaluated on chess, a controlled testbed with exact rules and strong engine-based reference signals. The task involves predicting optimal moves from a given position (FEN) while generating a reasoning trace.

• Models: Qwen3-8B and DeepSeek-R1-Distill-Llama-8B.
• Baselines: SFT only, and SFT followed by accuracy-only GRPO (Outcome-only RL).
• Metrics:
  • Accuracy: Top-1 move accuracy, Elo rating, candidate coverage.
  • Reasoning Quality: Win-rate Mean Absolute Error (MAE), Principal Variation (PV) overlap, and internal logic consistency (whether the chosen move matches the highest claimed win rate).
  • Judges: Three independent LLM judges (GPT-4o, Claude Opus 4.6, Gemini 3 Flash) evaluated reasoning on relevance, completeness, clarity, and fluency.

Key Results

The experiments demonstrate that VPS successfully decouples accuracy gains from reasoning degradation:

• Accuracy vs. Reasoning Trade-off: Accuracy-only GRPO improved Top-1 accuracy (e.g., from ~0.21 to ~0.53) but sharply degraded reasoning quality. For DeepSeek-R1-Distill-Llama-8B, internal consistency dropped from 0.978 to 0.312, and win-rate error increased by 112%.
• VPS Performance: VPS matched the accuracy of accuracy-only GRPO while significantly improving reasoning. It reduced win-rate error by up to 30% and restored internal consistency to near-saturation levels (0.978–0.985).
• Judge Evaluation: At matched accuracy levels, process-supervised models were consistently preferred by all three LLM judges over accuracy-only models, showing substantial improvements in completeness, clarity, and fluency.
• Adaptive Weighting: Adaptive weighting outperformed fixed aggregation, particularly for models with stronger initial domain knowledge (1M SFT positions), by focusing training on the hardest subtasks (e.g., PV prediction).
• Reasoning Space Analysis: Without a structured prior, accuracy-only RL converged to budget-dependent shortcuts (e.g., repetitive enumeration or superficial heuristics) rather than principled multi-step reasoning. Longer reasoning traces in accuracy-only models often correlated with lower accuracy, suggesting confusion rather than depth.

Significance and Claims

The paper claims that VPS provides a scalable, efficient alternative to LLM-judge-based process supervision. By leveraging structured priors and deterministic verification, it enables direct supervision of intermediate reasoning steps without the cost or noise of learned reward models.

The authors conclude that deterministic verification of intermediate reasoning, combined with an adaptive curriculum over subskills, is an effective approach for training models that achieve strong task performance with faithful reasoning. The results suggest that outcome-only optimization is insufficient for ensuring sound reasoning, and that VPS can prevent the "reasoning deterioration" observed in standard RLVR settings. The framework is presented as a generalizable method for verifiable domains, with chess serving as a rigorous testbed for fine-grained verification.