Technical Summary: One-Way Policy Optimization for Self-Evolving LLMs

1. Problem Statement

The paper addresses the limitations of Reinforcement Learning with Verifiable Rewards (RLVR) in scaling the reasoning capabilities of Large Language Models (LLMs). While RLVR offers a promising paradigm for tasks with checkable outcomes (e.g., math, code), it faces two primary challenges:

1. Sparse and Binary Signals: Verifier rewards are typically available only at the sequence level, leading to high variance, low sample efficiency, and unstable optimization.
2. The "Force Reversal" Conflict in Reference-Guided Methods: To stabilize training, existing methods (e.g., KL-regularized RL, On-Policy Distillation) introduce a reference policy ($\pi_{ref}$) to provide dense token-level constraints. However, these methods impose symmetric constraints that penalize any deviation from the reference, regardless of whether the deviation improves the reward.
   • The Core Issue: When a policy attempts to outperform the reference (a "Superior deviation"), the symmetric KL penalty or distillation loss exerts a "pull-back" force toward the reference. If this force is strong enough, it flips the optimization direction, turning a reward-improving step into a step backward. This creates a performance ceiling, preventing the model from evolving beyond the initial prior.

2. Methodology: One-Way Policy Optimization (OWPO)

The authors propose One-Way Policy Optimization (OWPO), a framework designed to decouple the optimization direction from the update magnitude.

Core Principle

• Direction: Dictated solely by the Verifier (via the advantage signal $A_t$). The verifier determines whether a token is correct or incorrect.
• Magnitude: Modulated by the Reference Policy ($\pi_{ref}$). The reference serves only to adjust the step size, not to reverse the direction.

Mechanism: Asymmetric Reweighting

OWPO introduces a dynamic scalar weight $w_t$ applied to the standard PPO/DAPO objective. This weight is determined by the Directional Deviation ($\delta_t$), defined as:
$$\delta_t(\theta) \triangleq \text{sgn}(A_t) \cdot \log \frac{\pi_\theta(y_t | s_t)}{\pi_{ref}(y_t | s_t)}$$
Based on $\delta_t$, OWPO applies two distinct regimes:

1. Accelerated Alignment (Inferior Deviations, $\delta_t < 0$):

   • Scenario: The current policy $\pi_\theta$ lags behind the reference in the correct direction (e.g., lower confidence on correct tokens or higher confidence on wrong tokens).
   • Action: The weight $w_t > 1$.
   • Effect: The gradient is amplified to accelerate the correction of these lags, leveraging the reference prior to fix errors quickly.

2. Gain Locking (Superior Deviations, $\delta_t > 0$):

   • Scenario: The current policy $\pi_\theta$ surpasses the reference in the correct direction (e.g., higher confidence on correct tokens).
   • Action: The weight $w_t < 1$ (specifically bounded by $\epsilon_{low}$).
   • Effect: The update step size is reduced. This "locks in" the sparse, high-value gains by reducing variance and preventing high-variance noise from destabilizing or reversing these beneficial deviations.

Iterative Self-Evolution (The Ratchet Effect)

To overcome the bottleneck of a fixed reference policy, OWPO incorporates an iterative bootstrapping process:

• At the end of each training stage, the reference policy is updated via a "Hard Swap": $\pi_{ref}^{(k+1)} \leftarrow \pi_\theta^{(k)}$.
• This creates a Ratchet Effect, where each iteration consolidates recent gains into a new baseline, allowing the model to continuously climb toward higher rewards without being constrained by the original prior.

3. Key Contributions

1. Identification of Direction Conflict: The paper identifies that symmetric constraints in existing RLVR methods cause "Force Reversal," where the reference policy inadvertently suppresses improvements that deviate from the prior.
2. Decoupling Direction and Magnitude: OWPO introduces a novel objective where the verifier dictates the gradient sign (direction), and the reference policy only scales the gradient magnitude.
3. Asymmetric Trust Region: Unlike standard symmetric KL regularization, OWPO constructs a direction-aware trust region that accelerates correction for inferior deviations while protecting superior deviations via variance reduction.
4. Iterative Self-Evolution Framework: By combining the one-way mechanism with periodic reference updates, OWPO enables continuous self-improvement without reliance on external teacher models.

4. Experimental Results

The authors evaluated OWPO on Qwen-2.5-Math-7B and Qwen-3-8B base models using the DAPO-Math-17k dataset and the AIME24/25 benchmarks.

• Performance vs. Baselines: OWPO consistently outperformed strong baselines, including pure RLVR methods (GRPO, DAPO) and reference-guided methods (OPD, MOPD, Sym-DAPO).
  • On Qwen-3-8B, OWPO achieved a Pass@1 of 51.90% (best) and 50.74% (converged) on AIME24, significantly surpassing MOPD (46.40%) and DAPO (45.31%).
  • OWPO broke the "performance ceiling" observed in MOPD and OPD, which tended to plateau near the reference policy's performance.
• Self-Evolution Efficiency: In an iterative setting starting from a suboptimal prior (~30% Pass@1), OWPO reached ~40% accuracy in 4 iterations, whereas MOPD required 6 iterations to saturate at a lower ~37%.
• Generalization: Despite training strictly on math data, OWPO demonstrated superior transferability on general science benchmarks (GPQA and MMLU-Pro) compared to other methods.
• Ablation Studies: Removing the asymmetry (Symmetric Reweighting) caused severe performance regression, confirming that symmetric penalties induce force reversal. Both "Accelerated Alignment" and "Gain Locking" components were found to be indispensable.

5. Significance and Claims

The paper claims that OWPO resolves the fundamental stability-exploration conflict in RLVR. By preventing the "Force Reversal" effect inherent in symmetric regularization, OWPO allows models to:

• Stably transcend suboptimal priors: Models can evolve beyond the capabilities of their initial reference policies without collapsing.
• Enable continuous self-evolution: The iterative "Ratchet Effect" allows for sustained performance improvement without external teacher models.
• Prioritize Verifier Guidance: The framework establishes that for verifiable reasoning tasks, the verifier's signal should be the primary driver of optimization direction, with reference policies serving only as stabilizers for magnitude.

The authors position OWPO as a fundamental step toward scalable, self-evolving reasoning models, noting that while it currently focuses on math and verifiable reasoning, the principle of direction-aware optimization is broadly applicable to RLVR.