Technical Summary: Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradients

Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a dominant paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs), particularly in mathematical domains. Current state-of-the-art approaches, such as Group Relative Policy Optimization (GRPO), rely on comparing grouped positive (correct) and negative (incorrect) rollouts to estimate advantages. However, this paper identifies a fundamental limitation in treating incorrect responses as explicit learning signals:

1. Lack of Gradation: In binary reward settings (e.g., $R(x,y)=1$ if correct, $0$ otherwise), all incorrect responses are treated equally. There is no inherent signal distinguishing a "slightly wrong" answer from a "completely nonsensical" one.
2. Combinatorial Vastness: The space of incorrect responses for complex tasks (like Olympiad-level math) is combinatorially vast. Penalizing a few sampled negative rollouts is unlikely to cover the meaningful failure modes or provide a robust reward signal under sparse binary rewards.
3. Inefficiency: The paper questions whether policy improvement can be achieved solely by reinforcing successes, drawing parallels to contrastive self-supervised learning (SSL) methods like BYOL and SimSiam, which learn effective representations using only positive samples.

Methodology: Positive-Only Policy Optimization (POPO)

The authors propose POPO, a novel on-policy RL framework that learns exclusively from positive rollouts (correct responses) while implicitly handling negative outcomes. The framework operates without disjoint negative rollouts for gradient guidance.

Core Components

1. Self-Normalized Importance Sampling:
   Instead of using a value network or group-relative advantage, POPO applies bounded importance sampling weights over the set of positive rollouts $S^+(x)$. The weight for a positive response $y$ is defined as:
   $$w_\theta(y|x) = \frac{\pi_\theta(y|x)}{Z_+(x)}, \quad \text{where } Z_+(x) = \sum_{y' \in S^+(x)} \pi_\theta(y'|x)$$
   This creates a self-competition mechanism where "confident" correct answers are reinforced more strongly, while normalization maintains diversity.

2. Implicit Negative Gradients:
   The paper provides a theoretical proof (Theorem 3.1) demonstrating that explicit negative penalties are unnecessary. Due to the softmax constraint ($\sum \pi_\theta(y|x) = 1$), increasing the probability of positive responses inherently decreases the probability of negative responses. The gradient analysis shows that the loss function induces a strictly positive gradient for incorrect logits $y' \in S^-(x)$, acting as an implicit "probability tax" and entropy-based penalty.

3. Adaptive Siamese Anchor:
   To stabilize training without a reference model (which can lead to distribution mismatch or overfitting), POPO employs a Siamese network architecture:

   • Policy Network ($\pi_\theta$): The active network being optimized.
   • Siamese Network ($\pi_\xi$): A target network updated via Exponential Moving Average (EMA) of the policy parameters: $\xi \leftarrow \tau \xi + (1-\tau)\theta$.
   • Representation-Space Alignment: Instead of standard KL-divergence (which can be high-variance and token-sensitive), POPO aligns the hidden representations of the policy and Siamese networks in a semantic embedding space using a predictor head and cosine similarity. This acts as a soft, bounded constraint.

4. Loss Function:
   The total loss combines the weighted negative log-likelihood (NLL) on positive samples, representation-space similarity, and entropy regularization:
   $$L_{POPO} = -\mathbb{E} \left[ \sum_{y \in S^+(x)} w_\theta(y|x) \log \pi_\theta(y|x) \right] + \alpha L_{sim} + \beta L_{ent}$$

Key Contributions

• Paradigm Shift: Introduces a framework where RLVR learning occurs exclusively via online positive rollouts, challenging the convention that negative samples are required for effective policy optimization.
• Theoretical Justification: Proves the existence of implicit negative gradients arising naturally from probability redistribution and entropy regularization, eliminating the need for explicit negative sampling.
• Stabilization Mechanisms: Proposes a Siamese architecture with momentum-based adaptation and representation-space similarity to replace KL-divergence, preventing policy collapse and unbounded drift.
• Bounded Importance Sampling: Utilizes a self-normalized weighting scheme over positive sets to balance reinforcement strength and diversity.

Experimental Results

The authors evaluated POPO using the DeepScaleR-Preview-Dataset for training and tested performance across five mathematical benchmarks: MATH-500, AMC 2023, AIME 2024, AIME 2025, and OlympiadBench.

• Models Tested: Experiments were conducted on various base models, including the Qwen2.5-Math family (1.5B, 7B), DeepSeek-R1 distilled models, and general-purpose LLMs (Llama-3.1-8B, DeepSeek-Math-7B).
• Performance:
  • POPO achieved performance comparable to or superior to GRPO and its variants (Dr. GRPO, DAPO, SAPO) across most benchmarks.
  • Notable Result: On the AIME 2025 benchmark using Qwen-Math-7B, POPO achieved 36.67%, outperforming GRPO (30.00%).
  • On OlympiadBench with Qwen-Math-7B, POPO reached 57.27%, compared to GRPO's 56.38%.
• Ablation Studies:
  • Removing negative rollouts (using uniform weights) caused a significant performance drop, confirming the necessity of the self-normalized weight redistribution.
  • Removing momentum adaptation or representation-space alignment led to instability and performance degradation, validating the stabilization mechanisms.
• Generalization: POPO demonstrated model-agnostic benefits, performing well on both math-specialized and general-purpose LLMs.

Significance and Claims

The paper claims to establish a new direction for RLVR that operates effectively without negative rollouts. The authors argue that in sparse binary reward environments, the combinatorial nature of failures makes explicit negative penalization inefficient and potentially noisy. By leveraging implicit negative gradients through probability redistribution and stabilizing the policy via representation-space alignment, POPO offers a robust alternative to current GRPO-based methods.

The authors note limitations, including the current focus on sparse binary rewards (though many real-world tasks have dense rewards), validation only on text-based mathematical reasoning, and testing limited to models up to 7B parameters. They suggest future work will explore adaptive rollout partitioning for negative samples and extension to code generation and multimodal reasoning.