Technical Summary: STRIDE - Learnable Stepwise Language Feedback for LLM Reasoning

1. Problem Statement

Recent advances in Large Language Model (LLM) reasoning have been driven by Reinforcement Learning from Verifiable Rewards (RLVR). However, existing approaches face two critical limitations:

1. Information Bottleneck of Scalar Rewards: Process Reward Models (PRMs) attempt to provide step-level supervision but rely on scalar scores. Compressing high-dimensional reasoning paths into single numerical values creates an information bottleneck where semantically distinct error modes (e.g., arithmetic errors vs. logical leaps) collapse into identical scores, offering insufficient semantic bandwidth to guide corrections.
2. Limitations of Existing Language Feedback: While critique-based methods offer richer textual feedback, they often rely on frozen or external critics, limiting scalability and adaptability. Furthermore, recent co-training frameworks (e.g., TANGO) that use generative verifiers often convert language output back into scalar rewards, reintroducing the same information bottleneck.
3. Credit Assignment: Outcome-based RL provides no feedback on individual reasoning steps, leaving the "credit assignment" problem (identifying which specific steps led to success or failure) largely unsolved in multi-step reasoning.

2. Methodology: The STRIDE Framework

STRIDE (Learnable Stepwise Language Feedback) proposes a paradigm shift from scalar process supervision to learnable, stepwise language feedback. It employs an interleaved three-phase training schedule involving a Generator ($G_\theta$) and a Generative Verifier ($V_\phi$), trained jointly using only outcome-based rewards (no step-level annotations).

Core Components

• Generator-Verifier Co-Training: Both models are optimized via Group Relative Policy Optimization (GRPO) using only the final outcome correctness ($c^*$). The verifier learns to decompose the terminal reward into step-wise linguistic critiques ($v_t$) without explicit step-level labels.
• Phase I: Base Policy Optimization: The generator is trained on outcome rewards to build fundamental reasoning capabilities. This phase occupies the majority of the training schedule (9/13).
• Phase II: Generative Verifier Optimization: The verifier is trained to generate high-fidelity language critiques ($v = (v_1, ..., v_T)$) for each reasoning step. It learns to localize errors and explain them linguistically, effectively acting as a "Contextual Navigator."
• Phase III: Guided Trajectory Redirection: This is the core innovation. When the generator fails, the verifier identifies the First Point of Failure (FPF), denoted as $t^*$. Instead of restarting from scratch, STRIDE employs a Multi-Point Redirection Strategy:
  • Anchors: It constructs redirection samples from all prefix steps $t \in [1, t^*]$, not just the FPF.
  • Instructions:
    • Rectification Prompt ($t=t^*$): The generator is instructed to correct the specific logical fallacy identified by the verifier.
    • Exploration Prompt ($t<t^*$): The generator is instructed to continue reasoning from a verified prefix, encouraging the discovery of alternative, more robust paths (addressing "latent drift" where early suboptimal choices lead to later failures).
  • Training Stability: Crucially, the advantage function in Phase III is calculated strictly based on the final outcome correctness of the redirected trajectory. This ensures that even if the verifier's language feedback is noisy or hallucinated, the policy update remains grounded in the verifiable outcome, preventing harmful gradient noise.

3. Key Contributions

1. Paradigm Shift to High-Bandwidth Feedback: The paper proposes shifting process supervision from unidimensional scalar rewards to learnable, in-context language feedback. This unlocks richer training signals capable of distinguishing qualitatively different error modes.
2. The STRIDE Framework: An interleaved three-phase co-training schedule that integrates a Multi-Point Redirection strategy. It utilizes verified semantic anchors to efficiently constrain the reasoning exploration space, allowing the model to learn from previously unsolvable problems.
3. Robustness to Verifier Noise: By grounding learning signals strictly to outcome correctness, STRIDE ensures "harmless policy improvement." Even with imperfect verifier localization, the framework guarantees non-negative improvement, as only successful redirections contribute to policy updates.

4. Experimental Results

Experiments were conducted on diverse benchmarks including mathematical reasoning (MATH-500, AIME 2024/2025, AMC 2023, OlympiadBench) and out-of-domain reasoning (BoardgameQA, CRUXEval, StrategyQA, TableBench).

• Performance: STRIDE significantly outperforms state-of-the-art baselines, including vanilla outcome-based RL (GRPO) and the scalar-reward co-training framework TANGO.
  • On Qwen2.5-7B, STRIDE achieved 84.6 on MATH-500 compared to TANGO's 81.4 and GRPO's 74.6.
  • On Llama-3.1-8B, STRIDE achieved 70.4 on MATH-500, outperforming TANGO (69.2) and GRPO (66.8).
  • Gains were consistent across logic, code, commonsense, and tabular reasoning tasks, indicating generalization beyond mathematics.
• Zero-Pass-Rate Problems: In ablation studies, STRIDE achieved a Correction Success Rate (CSR) of 6.8% on problems where all scalar-reward baselines yielded zero gradient signal (i.e., problems the model could not solve initially). This demonstrates the framework's ability to convert unsolvable instances into productive learning signals.
• Training Dynamics:
  • Verifier Quality: The verifier's step-level localization accuracy and critique helpfulness improved steadily during training, confirming that outcome-level supervision is sufficient to develop reliable process-level capabilities.
  • Exploration: STRIDE maintained consistently higher policy entropy than scalar-reward baselines, preventing premature convergence and representational collapse.
  • Reasoning Depth: Redirection phases fostered the emergence of deeper, more structurally complex reasoning chains that independent sampling could not produce.

5. Significance and Claims

The paper claims that STRIDE addresses the fundamental representational collapse inherent in scalar reward systems. By utilizing learnable stepwise language feedback, the framework provides the semantic direction necessary to localize and rectify specific reasoning errors.

The authors emphasize that STRIDE's effectiveness stems from the paradigm shift in supervision rather than increased computational budget. Compute-matched baselines (increasing training steps or batch size) showed negligible gains compared to STRIDE's performance leap. Furthermore, the framework demonstrates that high-bandwidth linguistic guidance can yield benefits even from imperfect verifiers, alleviating concerns about the reliability of learned critics. The work suggests that shifting from scalar to language-based process supervision is a viable path for overcoming reasoning plateaus in LLMs, enabling guided self-correction rather than exhaustive sampling.