Technical Summary: Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVR

1. Problem Definition: Correct-Set Turnover

The paper identifies a systematic limitation in Reinforcement Learning with Verifiable Rewards (RLVR) for Large Language Models (LLMs). While RLVR methods like GRPO and DAPO achieve headline accuracy gains, they often conceal a hidden cost: correct-set turnover.

This phenomenon is defined as the coupled dynamics of acquisition (newly solved problems) and regression (previously solved problems becoming unsolvable). The authors argue that standard RLVR pipelines treat forgetting merely as a side effect to be minimized. However, they propose viewing it symmetrically: as training proceeds, policy updates concentrate probability mass on newly reinforced solutions, causing the success probability of previously mastered prompts to drift downward due to gradient interference.

Key observations include:

• Asymmetry of Repair Cost: Drawing on spaced-repetition research (Ebbinghaus, 1885), the paper establishes that the cost of restoring a regressed prompt grows sharply with the delay of intervention. A "repair window" exists where a prompt is still partially mastered; reviewing it here is cheap. Once the prompt fully regresses, relearning it requires effort comparable to first-time learning.
• Limitations of Existing Methods: Standard pipelines revisit prompts only passively across epochs without tracking mastery status. Existing replay methods (e.g., ExGRPO, RLEP) reuse fixed historical trajectories but cannot detect if a previously mastered sample has regressed, rendering them ineffective against this specific type of intra-task forgetting.

2. Methodology: ReMind

To address correct-set turnover, the authors propose ReMind, a retention-aware review mechanism designed to operate within the standard RLVR training loop with zero additional rollout overhead.

Core Components

1. Graded Mastery Metric: Unlike binary "correct/incorrect" signals, ReMind utilizes the per-prompt success rate ($\hat{q}_t(p)$) derived from $K$ rollouts (already computed for advantage estimation in GRPO). A sample is considered "mastered" if $\hat{q}_t(p) = 1$.
2. Review Queue (FIFO):
   • When a sample from the standard training stream achieves mastery, it is appended to a FIFO review queue with probability $p_{add}$.
   • The queue stores only sample identifiers; fresh rollouts are generated from the current policy during review, avoiding the storage of stale trajectories.
3. Pre-Rollout Batch Replacement:
   • On designated review steps (every $f$ steps), a fraction $r$ of the training batch is replaced with samples drawn from the review queue.
   • This ensures the model is reminded of previous solutions before they leave the low-cost repair window.
   • Because review samples replace rather than augment the batch, the total rollout budget remains identical to the baseline.
4. Self-Correcting Update Rule:
   • Newly Mastered: Enqueued.
   • Retained: If a reviewed sample remains fully correct, it is removed from the queue (retention confirmed).
   • Regressed: If a reviewed sample fails ($\hat{q}_t(p) < 1$), it is reinserted at the tail of the queue for future repair.

3. Key Contributions

The paper makes three primary contributions:

1. Formalization of Correct-Set Turnover: The authors define the joint dynamics of acquisition and regression in RLVR and establish the repair-window principle: the cost of repairing regressed prompts increases non-linearly with intervention delay.
2. ReMind Algorithm: A lightweight, zero-overhead mechanism that maintains a mastery-keyed review memory and uses pre-rollout replacement to detect and repair regression without increasing the rollout budget.
3. Empirical Validation: Demonstration that ReMind improves late-stage accuracy and retention over standard RLVR baselines (GRPO, DAPO) and replay methods across 20 benchmarks spanning image-text, video, and text-only reasoning.

4. Experimental Results

The authors evaluated ReMind using Qwen3-VL-8B-Instruct (multimodal) and Qwen2.5-7B-Math (text-only) across 20 benchmarks.

• Performance Gains: ReMind consistently outperforms GRPO, DAPO, and replay baselines (ExGRPO, RLEP, RePO).
  • On image-text reasoning, ReMind improved the average accuracy by +3.81% over GRPO and +3.44% over DAPO.
  • Gains were most pronounced on math-heavy benchmarks (e.g., +5.67% on Math Reasoning average), supporting the hypothesis that longer reasoning chains are more fragile to policy drift.
• Mechanism of Improvement: Analysis of training dynamics (Figure 5) reveals that ReMind does not accelerate the acquisition of new solutions (both GRPO and ReMind solve new samples at similar rates). Instead, the improvement is driven entirely by reduced forgetting. ReMind reduced the accumulation of forgotten samples by approximately 35 compared to standard GRPO.
• Entropy Stability: ReMind mitigates the entropy collapse observed in standard GRPO, where the policy concentrates probability mass on fewer generation paths. Periodic review sustains policy entropy, preserving diverse solution strategies.
• Budget Sensitivity: The method is robust across a wide range of review budgets. Even a 1% review budget is sufficient to break the acquisition-regression stalemate, while performance slightly declines at very high budgets (>4%) due to the displacement of fresh training signals.
• Cross-Modal Generalization: The benefits extend to video understanding and pure text math reasoning, confirming the mechanism addresses a modality-agnostic failure mode.

5. Significance and Claims

The paper concludes that a significant portion of the potential in RLVR is not just about "learning more" but "losing less."

• Systematic Cause of Plateaus: The authors identify correct-set turnover as a systematic cause of accuracy plateaus in RLVR, distinct from cross-task catastrophic forgetting in continual learning.
• Design Paradigm Shift: The work argues for treating sample-level retention as a primary design axis in reasoning RL, alongside reward design and exploration.
• Efficiency: ReMind achieves these gains with negligible computational overhead (only ~3% of wall-clock time), requiring no additional model replicas or complex buffers.
• Intrinsic Nature of Regression: The authors modestly note that while ReMind reduces regression, it does not eliminate it entirely. Some degree of correct-set turnover appears intrinsic to the RLVR optimization process, suggesting that residual regression is a fundamental characteristic of the method rather than a transient artifact.

The paper does not claim to solve all forgetting issues or eliminate the need for careful reward design; rather, it provides a targeted, low-cost mechanism to stabilize the "correct set" during training.