Technical Summary: Rollout-Level Advantage-Prioritized Experience Replay for GRPO

Problem Statement

Reinforcement Learning from Verifiable Rewards (RLVR) using Group Relative Policy Optimization (GRPO) has become the standard for post-training reasoning in Large Language Models (LLMs). However, GRPO is inherently sample-inefficient. In the standard on-policy setting, a group of candidate rollouts is generated for each prompt, used for a single gradient update, and then discarded. This is particularly wasteful for rollouts that carry a strong learning signal, such as rare correct answers within a group of incorrect ones.

While Experience Replay (ER) is a common technique to improve sample efficiency in other reinforcement learning domains, it does not transfer directly to GRPO for two primary reasons:

1. Rapid Policy Drift: LLM policies shift significantly per gradient step. Stored rollouts become "stale" quickly, and using them without control can destabilize training.
2. Group-Level Granularity: Standard GRPO aggregates per-rollout learning signals into a single query-level value. Naive replay schemes that store and sample entire groups (query-level) fail to isolate the specific high-signal rollouts within a group that drive learning.

Existing attempts to apply replay to GRPO often suffer from unbounded staleness or rely on query-level priority signals (e.g., group variance) that cannot resolve the within-group minority rollouts where the most informative gradients reside.

Methodology

The authors propose a Rollout-Level Advantage-Prioritized Experience Replay buffer specifically designed for GRPO. The method integrates three core components to address staleness and signal isolation:

1. Age Eviction (Staleness Control)

To bound the policy lag of reused data, the buffer employs age eviction. Each stored rollout is tagged with a "birth step" ($t_b$). At any training step $t$, the age of a rollout is $t - t_b$. Any rollout older than a maximum threshold $\tau_{max}$ is evicted.

• Unlike capacity-based eviction, which varies with model scale and data difficulty, age eviction provides a step-level bound on staleness that is invariant to model size.
• This ensures that no update reuses a rollout beyond a fixed number of training steps, mitigating the risk of training on significantly off-policy data.

2. Fresh-Anchored Composition

The authors reject the standard "pool composition" (where fresh and replayed data are mixed into a single batch) in favor of fresh-anchored composition.

• At each training step, the full batch of fresh, on-policy rollouts is retained.
• A separate set of replay rollouts is sampled from the buffer and concatenated to the fresh batch.
• This ensures that every gradient update includes a "fresh anchor," bounding the worst-case policy lag independently of the replay portion. This is critical at larger model scales where mixing stale data with fresh data in a single pool was observed to degrade performance below standard GRPO.

3. Rollout-Level $|A_i|$ Priority

Instead of sampling entire groups (query-level), the buffer stores and samples individual rollouts.

• Priority Signal: The priority for sampling is the absolute advantage magnitude $|A_i|$ of the individual rollout. In GRPO, $A_i$ is the group-relative advantage. Rollouts with large $|A_i|$ (typically the minority in skewed groups, e.g., the single correct answer in a group of wrong ones) provide the bulk of the learning signal.
• Mechanism: By sampling at the rollout level, the method can selectively recycle these high-signal minority rollouts that are often discarded after a single pass in standard GRPO. This contrasts with query-level signals like group variance ($\sigma_g$), which treat all rollouts in a group equally and fail to prioritize the specific minority instances.

The training procedure (Algorithm 1) involves generating fresh rollouts, filtering out zero-variance groups, sampling replay rollouts based on $|A_i|$ priority from the age-gated buffer, and performing a GRPO update on the concatenated batch.

Key Contributions

1. Novel Replay Architecture: The paper introduces a GRPO-specific replay buffer combining age eviction, fresh-anchored composition, and rollout-level $|A_i|$ priority.
2. Empirical Superiority: The method outperforms standard GRPO and naive replay baselines (uniform and $\sigma_g$ priority with pool composition) across three model scales (0.6B, 1.7B, and 4B) on five math reasoning benchmarks.
3. Scaling Gains: Performance gains scale with model size. The largest improvement is observed at the 4B scale, with a +4.35 percentage point (pp) increase on the five-benchmark average compared to GRPO.
4. Efficiency Analysis: Using the Accuracy-Efficiency Score (AES), which jointly measures accuracy and token generation length, the method shows a significant efficiency margin over GRPO, particularly at the 4B scale (+0.579 AES).
5. Ablation Studies: Per-axis ablations validate that each component is necessary:
   • Age Eviction: A moderate cap ($\tau_{max}=10$) balances replay volume and staleness; larger caps lead to performance drops due to policy lag.
   • Composition: Fresh-anchored composition is essential at scale; pool composition causes the 4B model to fall below standard GRPO.
   • Priority: Rollout-level $|A_i|$ priority significantly outperforms query-level signals ($\sigma_g$ or uniform), particularly in improving pass@k (sampling diversity).

Results

• Accuracy: On the five-benchmark average (MATH-500, AIME25, AIME26, HMMT-F25, HMMT-F26), the 4B model achieved 31.06% accuracy with the proposed method, compared to 26.71% for standard GRPO and 26.70% for the best naive baseline.
• Sampling Diversity: The method improves pass@k metrics, indicating it retains the diversity of sampling that standard RLVR often narrows. At 4B, the method exceeded GRPO by +10.06 pp on the pass metric.
• Token Efficiency: The AES analysis reveals an asymmetric change in response length: the model generates longer responses for correct rollouts (+34.9% at 4B) and shorter responses for incorrect rollouts (-10.2% at 4B). This suggests the method optimizes the token budget by allocating more tokens to high-probability correct solutions while compressing incorrect attempts.

Significance and Claims

The paper claims that its method provides a sample-efficient training procedure for GRPO on verifiable-reward math reasoning tasks. The significance lies in demonstrating that experience replay can be successfully adapted to the unique constraints of LLM post-training (rapid drift and group-relative advantages) without destabilizing training.

The authors modestly frame their contributions as a specific solution to the sample inefficiency of GRPO, noting that:

• The method is currently limited to binary verifiable rewards (math reasoning) and does not claim generalization to non-verifiable rewards (e.g., learned reward models) or other RL algorithms (e.g., PPO, REINFORCE) without further study.
• The scaling trends are observed within the tested range (0.6B to 4B) and on specific prompt sets; transfer to larger scales or non-math domains is not validated.
• The results are based on a single training seed and specific evaluation settings, though the use of multiple benchmarks provides an indirect indication of stability.

Ultimately, the work establishes that rollout-level prioritization combined with strict staleness control allows for the effective recycling of high-value learning signals in GRPO, leading to substantial gains in both accuracy and token efficiency as model size increases.