Technical Summary: Reward Granularity in RLVR for Small Language Models

Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a paradigm for enhancing mathematical reasoning in language models, typically relying on outcome-based rewards (final answer correctness). However, the impact of process-level supervision (step-by-step rewards) remains underexplored, particularly for small language models (SLMs) like Qwen2.5-0.5B. These smaller models often lack the internal capacity to self-correct when provided with sparse, outcome-only feedback, leading to reasoning traces that may be logically inconsistent, hallucinated, or structurally unsound even when the final answer is correct. The central research question is whether process rewards improve the coherence and validity of reasoning traces and whether hybrid reward structures yield more stable reasoning than either approach alone.

Methodology

The authors conducted a controlled empirical study using Qwen2.5-0.5B, a compact transformer model, fine-tuned on the GSM8K dataset (grade-school math word problems) using Group Relative Policy Optimization (GRPO).

Experimental Setup

• Baseline: A pre-trained model evaluated in a direct-answer (no Chain-of-Thought) setting, achieving 33.13% accuracy.
• Reward Regimes: Five distinct reward conditions were compared:
  1. Process-Only: Rewards assigned to every intermediate step if it matches the ground-truth calculation (within numerical tolerance). A penalty is applied if the generated chain exceeds 1.5× the ground-truth step count.
  2. Outcome-Only: Binary reward based solely on the correctness of the final numerical answer.
  3. Hybrid Configurations: Weighted combinations of process and outcome rewards ($R = \lambda \cdot R_{process} + (1-\lambda) \cdot R_{outcome}$) with $\lambda \in \{0.9, 0.5, 0.1\}$.
• Evaluation Strategy:
  • Accuracy: Measured on the full 1,319-question test set.
  • Reasoning Fidelity: Evaluated on a stratified 45-question slice (15 easy, 15 medium, 15 hard) using four metrics:
    • CoT-Pass@1: Success rate of a single sampled trace.
    • Trace Validity: Logical well-formedness (absence of contradictions or invalid transformations).
    • Chain Length Deviation: Difference between generated and ground-truth step counts.
    • Process Step Ratio: Proportion of steps aligning with ground-truth reasoning.
  • Error Analysis: GPT-4o was used as a judge to categorize failure modes (e.g., incorrect derivation, calculator errors, missing steps, contradictions).

Key Results

1. Accuracy Gains

Process-only supervision achieved the highest test accuracy at 63.73%, representing a +30.60% improvement over the baseline. This significantly outperformed Outcome-only supervision (53.75%, +20.62% improvement).

• Hybrid Performance: The hybrid configuration with high process weight ($\lambda=0.9$) achieved 61.10% accuracy.
• Anomaly: The low-process/high-outcome configuration ($\lambda=0.1$) underperformed pure outcome supervision (49.30%), suggesting that conflicting optimization signals in this specific weighting degrade performance.

2. Reasoning Fidelity and Stability

• Trace Validity: The $\lambda=0.9$ hybrid model achieved the highest trace validity (0.60), followed by outcome-only (0.40) and process-only (0.22).
• Chain Length: Process-only models exhibited the largest deviation from ground-truth chain lengths (6.49), indicating a tendency toward verbose or redundant reasoning. Outcome-only models produced more concise traces (deviation 3.64).
• Process Step Ratio: The $\lambda=0.9$ hybrid model achieved the highest ratio (0.84), indicating the best alignment with ground-truth intermediate steps.

3. Qualitative and Error Analysis

• Process-Only Models: Generated structurally inconsistent but arithmetically grounded traces. They frequently introduced structural inconsistencies and contradictions, likely due to the incentive to produce long chains of locally plausible steps.
• Outcome-Only Models: Produced concise chains but were prone to derivation and arithmetic errors, reflecting weak intermediate grounding and a tendency to "jump" between reasoning points.
• Hybrid Models: Exhibited the most human-like reasoning flow, balancing step continuity with necessary brevity, and showed the most even distribution of error types with fewer catastrophic failures.

Significance and Contributions

The paper claims that reward granularity is a first-order design decision for RLVR, rather than a minor implementation detail. The key contributions are:

1. Systematic Comparison: Providing a controlled comparison of five reward conditions on a small model, demonstrating that dense process signals provide more informative guidance than sparse outcome feedback for SLMs.
2. Beyond Accuracy: Establishing that accuracy alone is insufficient; process supervision substantially improves trace fidelity and step validity, while outcome-only supervision can lead to "flaky" reasoning.
3. Hybrid Dynamics: Identifying that while hybrid rewards generally correlate positively with process weight, specific low-process configurations can create conflicting signals that harm performance.
4. Failure Mode Characterization: Using LLM-as-a-judge to reveal distinct failure distributions: process models suffer from structural inconsistency, while outcome models suffer from derivation errors.

The authors conclude that to develop models that not only answer correctly but also reason reliably, it is essential to supervise both the process and the outcome of reasoning. The nearly 10-percentage-point accuracy gap between process-only and outcome-only supervision underscores the critical importance of reward granularity in small language models.