Technical Summary: Correction-OrIented Policy Optimization (CIPO)

1. Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has become a standard paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs) in domains like mathematical reasoning and code generation. However, existing RLVR algorithms, such as Group Relative Policy Optimization (GRPO), face two critical limitations:

1. Sparse and Ambiguous Signals: RLVR relies on binary, sequence-level rewards (0 or 1). This sparsity results in weak credit assignment. Failed trajectories are uniformly penalized regardless of their "distance" from the correct answer (e.g., a minor calculation error vs. a fundamental logical flaw). This provides ambiguous optimization guidance, merely suppressing the likelihood of entire trajectories without offering direction on how to correct specific errors.
2. Underutilization of Failure: Failed trajectories often contain partially correct reasoning steps that constitute valuable learning signals. Current approaches discard these intermediate structures, wasting useful supervision and potentially hindering effective exploration.
3. Limitations of Existing Solutions: Previous attempts to address this, such as integrating Process Reward Models (PRMs) or LLM-based critics, often require additional human annotations, incur high computational costs, or introduce noise that undermines generalizability. Other methods relying on self-generated feedback may struggle with weaker models or suppress epistemic uncertainty.

There is a need for a task-agnostic solution that transforms failed trajectories into useful supervision without relying on external signals or additional annotations.

2. Methodology: Correction-OrIented Policy Optimization (CIPO)

CIPO is proposed as a systematic extension to the RLVR paradigm. Its core insight is that failed trajectories can be transformed from mere objects of penalty into exploitable supervisory signals by conditioning the model on its own erroneous outputs to generate corrections.

2.1 Core Framework

CIPO operates through an iterative cycle of generation and correction-oriented replay, optimizing two data streams simultaneously:

• Base Stream: Standard on-policy rollouts generated from original queries.
• Correction Stream: Refinement rollouts generated by conditioning the policy on the original query and a previous trajectory (the model's own attempt, whether correct or incorrect).

By sampling refined solutions ($y'$) conditioned on an erroneous attempt ($y_{fail}$), CIPO establishes a distinct gradient path connecting the specific failure mode to the goal state. This converts ambiguous suppression signals into precise, directional guidance.

2.2 Key Strategies for Stability and Efficiency

To prevent policy degradation and ensure efficient learning, CIPO integrates three specific mechanisms:

1. Adaptive Replay with Risk-Averse Shaping:

   • Adaptive Replay Ratio: A dynamic mechanism ($\rho_t$) adjusts the mixture of successful and failed trajectories in the replay buffer. If the model's performance on recycled successful samples degrades, the system increases the fraction of successful trajectories to stabilize learning.
   • Risk-Averse Reward Shaping: To prevent capability regression (where a model conditioned on a correct trajectory generates an incorrect response), an asymmetric penalty is applied. If the conditioning trajectory is correct but the new response is incorrect, an additional penalty is imposed. This prioritizes preserving existing correct behaviors while acquiring new ones.

2. Difficulty-Aware Trajectory Preference:

   • To optimize training efficiency, CIPO prioritizes replaying prompts with moderate pass rates (the "zone of proximal development"). Prompts that are consistently solved (too easy) or consistently failed (too hard) provide zero or low-value gradient signals. The system filters for medium-difficulty samples to ensure the model focuses on the effective learning window.

3. Joint Optimization Objective:
   The final objective function combines the standard GRPO loss on base rollouts with a correction loss on the replayed rollouts. The correction rewards incorporate the risk-averse shaping. The model is trained to maximize the expected reward across both streams, effectively learning to correct its own errors while maintaining standard reasoning performance.

3. Key Contributions

• Revisiting Failure in RLVR: The paper investigates how on-policy failed trajectories can be converted from sparse negative feedback into useful, correction-oriented supervision without external models.
• CIPO Algorithm: A novel extension to RLVR that constructs correction samples from the model's own on-policy failures. It eliminates the need for external annotations or auxiliary reward models.
• Stability Mechanisms: The introduction of adaptive replay ratios and risk-averse reward shaping to balance exploration (learning from failure) with exploitation (preserving correct capabilities).
• Empirical Validation: Extensive experiments demonstrating that CIPO improves both reasoning and error-correction capabilities, with gains in pass@K metrics suggesting a genuine expansion of intrinsic reasoning capacity rather than mere probability redistribution.

4. Experimental Results

The authors evaluated CIPO across 11 benchmarks spanning mathematical reasoning and code generation, using Qwen3-4B for math and Seed-Coder-8B for code.

• Reasoning Performance: CIPO consistently outperformed strong baselines (including vanilla GRPO with various batch sizes and PRIME).
  • On mathematical reasoning, CIPO achieved an average accuracy of 64.38%, surpassing the best GRPO baseline by 4.55%.
  • Significant gains were observed on challenging datasets like AIME24 and AIME25.
• Correction and Debugging:
  • On DebugBench, Seed-Coder-8B trained with CIPO achieved a 7.63% gain, reaching performance comparable to Claude-4-sonnet and surpassing GRPO.
  • On CriticBench, the model showed improved correction rates (e.g., +7.74% on Math tasks), demonstrating robust cross-scenario generalization to symbolic and algorithmic reasoning tasks.
• Intrinsic Capacity Expansion:
  • CIPO yielded higher pass@K scores (e.g., pass@32 on math benchmarks), indicating that the model's intrinsic reasoning capacity was genuinely expanded rather than just redistributing probability mass over existing correct answers.
  • Training dynamics showed a robust, monotonic upward trajectory, avoiding the performance saturation and fluctuation observed in GRPO baselines.

5. Significance and Claims

The paper claims that CIPO represents a significant step forward in making RLVR more efficient and effective by fully utilizing the information embedded in failed trajectories.

• Directional Guidance: Unlike standard RLVR which uniformly suppresses failures, CIPO differentiates failure modes. It leverages the fact that "near-miss" attempts have a higher probability of yielding correct solutions during refinement, thereby extracting richer, denser learning signals.
• Self-Correction Capability: By explicitly training the model to generate correct solutions conditioned on its own erroneous attempts, CIPO enhances the model's ability to debug and refine, extending its practical applicability.
• Task-Agnostic and Efficient: The method requires no external signals, human annotations, or additional reward models, preserving the simplicity and scalability of the RLVR paradigm while addressing its fundamental limitations regarding sparse rewards.

The authors conclude that CIPO effectively transforms the "failure" of a trajectory into a "success" of the learning process, enabling models to learn from their own mistakes in a stable and scalable manner.