GRPO, Dr. GRPO, and DAPO Are Three Operations on One Number: The Group-Standard-Deviation Identity

Technical Summary: The Group-Standard-Deviation Identity in RLVR Training

Problem Statement
In Reinforcement Learning with Verifiable Rewards (RLVR), training language models to reason often involves sampling multiple candidate answers ($G$) for a single prompt and using an automatic verifier to assign binary rewards (1 for correct, 0 for incorrect). Three prominent methods for processing these rewards—Group Relative Policy Optimization (GRPO), GRPO Done Right (Dr. GRPO), and Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO)—appear to employ distinct heuristics. GRPO normalizes advantages by the group's reward standard deviation; Dr. GRPO removes this normalization; DAPO discards groups where all answers agree (zero standard deviation). The paper addresses the lack of a unified theoretical understanding of these methods, questioning whether they are separate fixes or variations of a single underlying mechanism.

Methodology and Core Identity
The paper isolates the training update for a single prompt with a finite group size $G$ and binary rewards. It derives an exact mathematical identity for the per-prompt policy gradient update, independent of the baseline used.

Let $k$ be the number of correct answers in a group of size $G$. The group reward standard deviation is $\sigma = \sqrt{k(G-k)/G}$. The paper proves the Group-Standard-Deviation Identity:
$$g = \sigma (\bar{s}_+ - \bar{s}_-)$$
where $\bar{s}_+$ and $\bar{s}_-$ are the mean scores of the correct and incorrect rollouts, respectively.

This identity reveals that the magnitude of the training update is exactly the group's reward standard deviation. The update vanishes when the group is unanimous ($k=0$ or $k=G$, so $\sigma=0$) and peaks when the group is evenly split ($k=G/2$).

Unification of Methods
The paper demonstrates that GRPO, Dr. GRPO, and DAPO are not distinct algorithms but three different operations applied to this single scalar quantity ($\sigma$):

1. GRPO: Divides the advantage by $\sigma$. This scales the update by $\sigma$, effectively optimizing a variance-stabilized objective ($2\arcsin\sqrt{p}$).
2. Dr. GRPO: Removes the division by $\sigma$. This results in an update proportional to $\sigma^2$, optimizing the raw success rate ($p$).
3. DAPO: Discards groups where $\sigma = 0$. This filters out "silent" groups that provide no contrast between right and wrong answers.

Key Contributions
The paper provides five specific theoretical and empirical contributions:

1. The Exact Finite-Group Identity: The paper establishes that the GRPO update is exactly the contrastive score direction scaled by $\sigma$, valid for any finite $G$ and any policy dimension, without relying on asymptotic limits.
2. The Group-Size Law: It derives a closed-form relationship determining how many samples ($G$) are required to achieve a target fidelity ($\phi$) of the large-group gradient. The required group size scales with the square of the difficulty weight: $G \gtrsim 1/(8\epsilon p(1-p))$. This implies that hard prompts (low $p$) require significantly larger group sizes to produce a stable learning signal compared to medium-difficulty prompts.
3. The Silent-Group Rate: The paper quantifies the probability of a group being "silent" (providing no gradient) as $P = p^G + (1-p)^G$. This explains the mechanism behind DAPO's dynamic sampling, which over-samples and discards these specific groups.
4. Exact Characterization of Difficulty Bias: The paper proves that the "difficulty bias" observed in GRPO (where easy and hard prompts are over-weighted) is a direct consequence of the $1/\sigma$ division. This division corresponds to the derivative of the arcsine transform ($\partial_p 2\arcsin\sqrt{p} = 1/\sqrt{p(1-p)}$), which assigns higher marginal weight to extreme difficulties. Removing the division (Dr. GRPO) reverts the objective to the raw success rate with flat difficulty weighting.
5. Empirical Validation:
   • Big-Math Dataset: On a dataset of 215,608 problems, the paper shows that GRPO's standardization shifts the gradient budget, allocating nearly double the share to extreme-difficulty prompts (13.9% $\to$ 24.7%) compared to Dr. GRPO. It also confirms that at standard group sizes ($G=8$), 44% of prompts yield silent groups, matching the theoretical prediction.
   • Controlled Training Run: A reproducible experiment with 6,000 Bernoulli-logit prompts confirms that the closed-form predictions for silent-group rates and gradient mass distribution track the actual training dynamics with high precision ($R^2 = 0.999$ for silent groups).

Significance and Claims
The paper claims that what is often viewed as a cosmetic normalization step is actually the "dial" that determines where and how strongly learning occurs.

• Signal vs. Noise: The group standard deviation is not merely a normalizer; it is the size of the learning signal itself. A prompt teaches only when its sampled answers disagree.
• Unified View: The three methods are unified as operations on one number. GRPO chooses to amplify the signal by the disagreement level; Dr. GRPO chooses to ignore the disagreement level; DAPO chooses to ignore the cases where there is no disagreement.
• Practical Implications: The findings suggest that using a uniform group size is suboptimal, as it underserves hard prompts (which need larger $G$ to avoid silence) and overserves medium prompts. The paper argues that understanding the exact finite-group accounting allows practitioners to make informed choices about group size and objective formulation based on the specific difficulty distribution of their data, rather than relying on asymptotic approximations or heuristic tuning.

The paper concludes that these results do not require new algorithms but follow from exact accounting of a single sampled group, providing a closed-form framework for analyzing reward shaping in RLVR.