Technical Summary: Exploration vs Exploitation in RLVR

1. Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a primary driver for enhancing the reasoning capabilities of Large Language Models (LLMs), particularly in mathematical domains. However, recent empirical observations present a paradoxical dynamic: performance improvements in RLVR have been attributed to two mechanisms that seemingly contradict standard reinforcement learning (RL) intuitions regarding the exploration-exploitation trade-off:

1. Spurious Rewards: Using random or misaligned rewards (e.g., Bernoulli(1/2) signals independent of ground truth) has been observed to improve performance in certain models (e.g., Qwen-Math), whereas classical RL theory suggests such noise should hinder exploitation.
2. Entropy Minimization: Reducing policy entropy to encourage deterministic, high-confidence outputs has been linked to gains, despite the standard view that entropy reduction leads to premature convergence and suboptimal policies.

The paper investigates the underlying principles reconciling these effects, specifically questioning whether spurious rewards provide meaningful learning signals, how clipping bias interacts with policy entropy, and whether the observed gains are merely artifacts of data contamination or model-specific quirks.

2. Methodology and Theoretical Framework

The authors employ a combination of theoretical analysis and extensive empirical evaluation across multiple model families (Qwen-Math, Llama, QwQ) and sizes (7B to 32B).

2.1 Theoretical Analysis of Clipping and Entropy

The paper analyzes Group Relative Policy Optimization (GRPO), the standard algorithm for RLVR.

• Clipping Bias: The authors derive explicit bounds on the magnitude of the upper-clipping correction term ($C^+_{tot}$). They prove that under random rewards, the clipping bias does not constitute a meaningful learning signal. Theoretical bounds (Theorem 3.2, Theorem 3.4) demonstrate that the magnitude of the raw surrogate sum ($N_{raw}$) significantly exceeds the clipping correction term ($E[|N_{raw}|] \gg E[|C^+_{tot}|]$), implying clipping does not drive performance gains directly.
• Entropy Dynamics: The paper challenges the approximation in prior work (Cui et al., 2025) which suggests entropy remains constant under random rewards. By incorporating clipping and higher-order terms, the authors derive a "one-step policy-entropy shift" formulation.
  • Unclipped Training: Entropy change depends on the initial policy skewness ($\Phi(\pi_{old})$). If the policy is sufficiently skewed, entropy can increase; otherwise, it decreases.
  • Clipped Training: The authors prove (Theorem 4.3) that upper clipping acts as an implicit entropy minimization mechanism. Clipping systematically reduces policy entropy, driving the policy toward more deterministic, higher-confidence rollouts.

2.2 Reward Misalignment Model

To explain why random rewards might benefit some models but not others, the authors introduce a probabilistic reward-misalignment model. They define "damage" ($\Delta$) as the loss of advantage mass for correct rollouts due to random mislabeling (False Positives and False Negatives).

• Model Strength Dependency: The analysis shows that the expected damage and its variance decrease as the number of correct rollouts ($n_c$) increases. Consequently, stronger models (where $n_c > n_i$) experience less total misalignment damage and more stable training dynamics, making them more likely to benefit from random rewards.
• Contamination Hypothesis: The model suggests that improvements are not solely due to data contamination (memorization of validation sets) but are a broader phenomenon dependent on the model's initial capability relative to the dataset difficulty.

3. Key Contributions

1. Theoretical Bounds on Clipping: The paper derives explicit bounds showing that under spurious rewards, clipping bias provides no meaningful learning signal. Instead, it establishes a deterministic link where clipping systematically reduces policy entropy.
2. Decoupling Entropy and Performance: The authors demonstrate that there is no direct causal relationship between policy entropy reduction and performance improvement. While entropy minimization can coincide with gains, it can also lead to performance collapse, particularly in weaker models or on difficult datasets.
3. Generalization Beyond Contamination: Through experiments on non-contaminated model families (Llama, QwQ) and varying dataset difficulties (DeepScaleR vs. AIME), the paper shows that performance gains under random rewards are robust and not confined to potentially contaminated Qwen-Math models.
4. Reward Misalignment Framework: A new theoretical model explains the conditions under which random rewards are beneficial, linking success to model strength and the ratio of correct to incorrect rollouts.

4. Experimental Results

• Clipping vs. Unclipped: In experiments with Qwen2.5-Math-7B on random rewards, disabling clipping led to an increase in policy entropy and, in some cases, performance improvement. Enabling clipping caused a monotonic decrease in entropy. Crucially, for the weaker Qwen-Math-7B model on the difficult AIME dataset, entropy minimization (via clipping) led to suboptimal performance, whereas stronger models (QwQ-32B, R1-Distill-Llama-8B) showed steady gains.
• Gradient Explosion: The authors observed that without clipping, models with large gradient norms (e.g., R1-Distill-Llama-8B) can suffer from gradient explosion and training collapse, highlighting clipping's role as a stability mechanism rather than a performance driver.
• Model Strength: Experiments confirmed that stronger models (e.g., Llama-8B distilled) improved under random rewards, while weaker models (e.g., Qwen-1.5B) degraded or remained unstable. This supports the reward-misalignment theory that gains are contingent on the model's ability to generate a sufficient number of correct trajectories.
• Hyperparameter Sensitivity: Varying the clipping ratio ($\epsilon$) and group size ($G$) showed that while stricter clipping reduces variance, it does not fundamentally alter the conclusion that clipping acts as an entropy regulator.

5. Significance and Claims

The paper claims to clarify the mechanisms behind the "spurious reward" phenomenon in RLVR, overturning the prevailing view that improvements are solely due to data contamination or that clipping bias provides a learning signal.

• Reframing Clipping: The authors assert that clipping in RLVR under random rewards functions primarily as a regularizer that enforces a local trust region and reduces policy entropy, rather than as a source of informative gradients.
• Context-Dependent Benefits: The paper concludes that the benefits of spurious rewards and entropy minimization are regime-dependent. They are effective for strong models on easier datasets where the policy is already concentrated on correct trajectories, but they can reinforce incorrect modes and degrade performance for weaker models or harder tasks.
• No Universal Entropy-Performance Link: The work explicitly refutes the idea of a monotonic relationship between lower entropy and higher performance, showing that entropy reduction can accompany performance collapse.

In summary, the paper posits that the success of RLVR with spurious rewards is not a paradox but a result of specific interactions between model strength, dataset difficulty, and the implicit regularization effects of clipping on policy entropy. These findings provide principles for more effective RLVR training, suggesting that entropy control should be viewed as a stabilizing regularizer rather than a universal objective for improvement.