Technical Summary: Unbounded Positive Asymmetric Optimization (UP)

1. Problem Statement: The Exploration-Stability Dilemma

Reinforcement learning (RL) has become the standard paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs). While modern frameworks like Group Relative Policy Optimization (GRPO) and Dynamic sAmpling Policy Optimization (DAPO) utilize Importance Sampling (IS) to improve sample efficiency, they face a fundamental exploration-stability dilemma.

• Instability of Pure IS: Pure, unclipped IS is prone to catastrophic training instability. When evaluating rare, long-tail reasoning paths, the importance sampling ratio ($r_{i,t}$) can explode if the historical policy ($\pi_{old}$) assigns infinitesimal probability to a correct action. This leads to pathological gradients that destroy stable representations.
• Stifling of Exploration by Clipping: To mitigate instability, standard algorithms employ clipping mechanisms (e.g., bounding the ratio within $[1-\epsilon, 1+\epsilon]$). However, the authors argue that this conservative clipping structurally stifles exploration.
• Probability Capacity (Cap): The paper formalizes the concept of Probability Capacity (Cap), defined as the maximum allowable increase in token probability for positive advantages before the gradient is truncated. The authors demonstrate that conservative clipping restricts Cap to be linearly proportional to the historical policy ($\pi_{old}$). For low-confidence but correct reasoning paths (where $\pi_{old}$ is small), the update budget is severely constrained (e.g., a mere 0.0028 increase), causing gradients to vanish prematurely. This prevents the model from reinforcing promising long-tail trajectories.

2. Methodology: Unbounded Positive Asymmetric Optimization (UP)

To resolve this dilemma, the authors propose Unbounded Positive Asymmetric Optimization (UP), a universal, plug-and-play objective that restructures the optimization process by treating positive and negative advantages asymmetrically.

Core Mechanisms

1. Self-Anchored Ratio via Stop-Gradient:
   For positive advantages (correct rollouts), UP replaces the standard IS ratio ($\pi_{new}/\pi_{old}$) with a self-anchored ratio using the stop-gradient operator ($sg$):
   $$\tilde{r}_{i,t}(\theta) = \frac{\pi_\theta(\cdot)}{sg(\pi_\theta(\cdot))}$$

   • Theoretical Implication: Because the denominator is treated as a constant during backpropagation, the gradient of this objective becomes mathematically equivalent to the stable, unclipped REINFORCE gradient ($\nabla \log \pi_\theta$).
   • Result: This formulation completely eradicates the root cause of IS-induced instability (the exploding $\pi_{old}$ in the denominator) while allowing unbounded updates. It effectively removes the clipping constraint for correct paths, maximizing the Probability Capacity to $1 - \pi_\theta$.

2. Asymmetric Design:

   • Positive Advantages ($\hat{A} > 0$): The unbounded, self-anchored formulation is applied to aggressively reinforce correct, low-confidence reasoning paths without clipping.
   • Negative Advantages ($\hat{A} \leq 0$): For incorrect rollouts, the method retains standard clipping mechanisms (e.g., from DAPO or GRPO). This acts as a structural safeguard to prevent the unbounded mechanism from causing aggressive gradient ascent on wrong actions, which would destroy the original representation.

Integration with Existing Frameworks

The UP framework is designed to be a universal plugin compatible with various optimization granularities:

• Token-Level: UP-DAPO and UP-GRPO replace the clipped positive branch with the unbounded log-policy objective while keeping the negative branch clipped.
• Sequence-Level: UP-GSPO extends this logic to sequence-level importance sampling, applying the unbounded formulation to the length-normalized sequence advantage.

3. Key Contributions

1. Formalization of Probability Capacity (Cap): The paper introduces Cap to mathematically expose the structural bottleneck in current RL algorithms, demonstrating how conservative clipping prematurely truncates updates for correct but low-confidence paths.
2. Theoretical Restructuring: By anchoring the policy to its current state via the stop-gradient operator, UP theoretically restructures the IS mechanism to bypass the trust region bottleneck for positive advantages, aligning unclipped gradients with REINFORCE stability.
3. Universal Plug-and-Play Objective: UP is validated as a universal enhancement applicable across different algorithms (DAPO, GSPO, GRPO), model architectures (Dense, MoE, Vision-Language), and training modalities (Language, Multimodal).

4. Experimental Results

Extensive experiments were conducted across diverse benchmarks (AIME24, AMC23, MATH500, Minerva, OlympiadBench, Geometry3K) and model families (Qwen3 series).

• Performance Gains:

  • UP-GRPO achieved the best average Pass@1 accuracy (61.31%) across five challenging reasoning benchmarks, outperforming 11 strong baselines including GSPO (60.15%), ASPO, and REINFORCE++.
  • UP-DAPO showed significant improvements over standard DAPO, achieving a peak Avg@32 of 51.15% vs. 47.71% on AIME24.
  • UP-GSPO improved the MoE model (Qwen3-30B-A3B-Base) by 3.02% absolute accuracy on AIME24.
  • UP-GRPO on Vision-Language models (Qwen3-VL-8B) improved accuracy on Geometry3K by 3.30%.

• Exploration and Stability:

  • Enhanced Exploration: UP variants maintained consistently rising policy entropy throughout training, whereas baselines (like GRPO, GSPO) exhibited entropy collapse. This indicates UP successfully preserves the ability to explore alternative reasoning trajectories.
  • Stability: Despite removing clipping for positive advantages, UP maintained gradient norms and KL divergence levels comparable to or lower than baselines. This confirms that the asymmetric design prevents training instability while maximizing exploration.

• Ablation Studies:

  • Simply removing the upper clip bound in DAPO (without the self-anchored ratio) led to immediate gradient explosion, proving the necessity of the stop-gradient operator.
  • Applying unbounded updates to both positive and negative advantages (symmetric unbounded) caused catastrophic instability, validating the necessity of the asymmetric design.

5. Significance and Claims

The paper claims that UP provides a mathematical foundation to break free from the conservative clipping constraints that have historically limited RL-based reasoning. By decoupling the treatment of correct and incorrect rollouts, UP allows models to aggressively internalize long-tail reasoning capabilities without sacrificing optimization stability.

The authors position UP not merely as a heuristic improvement but as a universal paradigm that resolves the exploration-stability dilemma inherent in IS-based RL. The results suggest that the "unbounded positive" approach is a critical, previously underexplored direction for enhancing the reasoning capabilities of LLMs across diverse architectures and modalities.