Technical Summary: Discovering Process–Outcome Credit in Multi-Step LLM Reasoning

1. Problem Statement

Reinforcement Learning (RL) has emerged as a potent paradigm for enhancing reasoning in Large Language Models (LLMs), particularly through frameworks like Reinforcement Learning with Verifiable Rewards (RLVR). However, standard outcome-based approaches, such as Group Relative Policy Optimization (GRPO), suffer from reward sparsity. In complex reasoning chains, binary feedback (correct/incorrect) provides no signal for intermediate steps, leading to inefficient credit assignment and difficulty in navigating long-horizon tasks.

Existing attempts to introduce dense rewards face a dual bottleneck:

1. Process Reward Models (PRMs) rely on labor-intensive, human-annotated rationales, hindering autonomous scaling.
2. Alternative dense signals (e.g., hand-crafted rules or auxiliary models) are often susceptible to reward hacking and exhibit a semantic disconnect, failing to accurately quantify the functional proximity between intermediate steps and the terminal goal.

The core challenge addressed is how to architect a learning framework that facilitates dense, step-wise semantic guidance and rigorous outcome alignment without resorting to exogenous expert demonstrations.

2. Methodology

The proposed framework, illustrated in Figure 1 of the paper, integrates three core mechanisms to synergize intrinsic exploration with strict outcome constraints.

2.1. Structured Rollout and Parsing

For each prompt, the policy generates a group of trajectories. Unlike outcome-only methods, these trajectories are parsed into discrete reasoning steps ($s_1, \dots, s_K$) using a predefined structural schema (e.g., ### Step i). This enables fine-grained credit assignment rather than assigning a single scalar reward to the entire chain.

2.2. Step-wise Marginal Information Gain (MIG)

To provide dense, intrinsic rewards, the authors introduce a Step-wise Marginal Information Gain (MIG) mechanism.

• Step-Conditioned Likelihood: For each step $s_k$, the method computes the length-normalized log-likelihood of the ground truth $y^*$ conditioned on the prefix generated so far ($\ell_k$). This serves as a proxy for the model's confidence in the correct outcome.
• Monotonic Historical Watermark (HWM): To prevent reward hacking via "pump-and-dump" oscillations (where a model degrades likelihood only to improve it later), a strict monotonicity constraint is enforced. A historical watermark $h_k$ tracks the maximum semantic fidelity achieved up to step $k$: $h_k = \max(h_{k-1}, \ell_k)$.
• Rectified Breakthrough Reward: The dense reward $g_k$ is defined as the rectified increment above this watermark: $g_k = \max(0, \ell_k - h_{k-1})$. This ensures rewards are position-agnostic; only steps that provide a genuine semantic breakthrough (increasing likelihood beyond the historical maximum) receive credit. Redundant steps receive zero reward.

2.3. Hybrid Optimization via Decoupled Masking

The framework employs a multi-objective loss function ($L = \alpha L_{MIG} + \beta L_{Outcome} + \gamma L_{Gated-SFT}$) with a Decoupled Masking Strategy to handle divergent exploration and convergent grounding:

• $L_{MIG}$ (Process Exploration): Optimizes the reasoning process using the dense MIG rewards, applied specifically to Chain-of-Thought (CoT) tokens. This encourages broad cognitive exploration.
• $L_{Outcome}$ (Outcome Grounding): Applies a standard GRPO-style loss to the full completion, combining a binary correctness reward with a format compliance reward. This enforces strict adherence to the correct answer and structural validity.
• $L_{Gated-SFT}$ (Stabilization): A dual-gated Self-Supervised Fine-Tuning objective. This loss is activated only when two conditions are met: (1) the trajectory follows a valid structural format, and (2) the final answer is correct. This ensures high-fidelity data distillation, preventing the accumulation of logical hallucinations while stabilizing training.

3. Key Contributions

The paper outlines four primary contributions:

1. Process Exploration via Step-wise Probabilistic Reward: A dense reward mechanism based on Marginal Information Gain (MIG) against a Monotonic Historical Watermark. This resolves the credit assignment problem by rewarding intrinsic semantic novelty regardless of sequential position, filtering out redundant oscillations without external annotations.
2. Outcome Grounding via Gated Self-Correction: A strategy that applies Supervised Fine-Tuning (SFT) exclusively when the model's exploratory trajectory converges to a verified correct outcome. This balances creative process exploration with strict outcome adherence.
3. Hybrid Reward Optimization with Decoupled Masking: A dual-objective scheme that separates the dense reward (for expanding the reasoning search space) from the sparse binary reward and gated SFT (for bounding optimization within the manifold of correctness).
4. Cross-Modal Scalability: The demonstration that the framework extends beyond text-only domains to complex multi-modal reasoning tasks, suggesting that intrinsic semantic novelty is a modality-agnostic driver of general intelligence.

4. Experimental Results

The authors conducted extensive experiments on eight datasets, including textual (GSM8K, MATH, SCQ5K) and multi-modal (Super-CLEVR, ChartQA, CMM-Math) benchmarks, as well as Out-of-Distribution (OOD) evaluations.

• Sample Efficiency and Convergence: The MIG framework demonstrated significantly higher sample efficiency than GRPO baselines. On structured tasks like GSM8K and Super-CLEVR, the model achieved near-saturation performance (approx. 95%) within the first 100 steps, whereas GRPO struggled with initial exploration.
• Robustness to Reward Hacking: On the SCQ5K-EN benchmark, where GRPO underperformed the base model (indicating overfitting to formatting), the proposed method improved performance by 11.0%, valuing the reasoning process itself.
• Generalization (OOD): The model exhibited superior transfer capabilities. On the MMStar benchmark, GRPO suffered from negative transfer (performance drop), while the proposed method achieved positive transfer (+5.0%).
• Hard Reasoning Capability: On the challenging AIME 2025 benchmark, the full model achieved a breakthrough of 13.3% accuracy, whereas the ablated model (without Gated-SFT) collapsed to 0.0%, highlighting the necessity of the SFT component for stabilizing complex reasoning.
• Ablation Insights:
  • MIG vs. Position Bias: The HWM mechanism outperformed position-weighted heuristics, confirming that pivotal insights can occur at any stage, not just late in the chain.
  • SFT Role: While pure RL (MIG) drove deep reasoning on the hardest problems (Level 5 MATH), the inclusion of Gated-SFT was critical for preventing foundational collapse on easier problems (Level 1) and ensuring overall stability.

5. Significance and Claims

The paper claims to address the fundamental challenge of aligning LLMs with sparse outcome rewards by introducing an efficient training framework that derives continuous reward signals from the model's own probability landscape.

• Modality Agnosticism: The consistent gains across textual and multimodal domains suggest that step-wise likelihood mechanisms are a universal proxy for logical progress, independent of input modality.
• Scalability: By extracting supervision directly from the model's internal probabilistic transitions rather than relying on expensive human annotations, the method offers a scalable pathway for the next generation of reasoning alignment.
• Robustness: The framework fosters deep, transferable reasoning capabilities that withstand distribution shifts, overcoming the limitations of standard RL methods that often collapse on olympiad-level or out-of-distribution tasks.

The authors conclude that by balancing intrinsic curiosity (via MIG) with strict correctness constraints (via Gated-SFT), their approach effectively bridges the gap between open-ended thinking and precise answering.