EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget

1. Problem Statement

The paper addresses a critical limitation in Reinforcement Learning with Verifiable Rewards (RLVR) for Large Language Models (LLMs), specifically when using Group Relative Policy Optimization (GRPO).

• The Core Issue: RLVR methods often suffer from entropy collapse. Due to the exploitative nature of reward maximization, policies rapidly converge to "dominant modes" (specific reasoning paths that yield high rewards).
• The Self-Reinforcing Loop: Once a dominant mode is identified, the policy repeatedly samples it during rollouts. Since these samples receive positive feedback, the policy further amplifies this mode while suppressing alternative strategies.
• Consequences:
  • Overfitting: The model performs well on in-distribution test sets but fails on Out-of-Distribution (OOD) benchmarks (e.g., AMC 2023).
  • Local Optima: The model gets stuck in suboptimal reasoning strategies, failing to discover potentially superior solutions.
  • Failure of Existing Fixes: Current methods like increasing sampling temperature or adding entropy regularization merely flatten the distribution uniformly. They fail to shift probability mass away from dominant modes effectively and often lead to instability or degraded performance.

2. Methodology: EEPO

The authors propose Exploration-Enhanced Policy Optimization (EEPO), a framework that disrupts the self-reinforcing loop via a "Sample-Then-Forget" mechanism during the rollout phase.

Core Mechanism: Two-Stage Rollout with Unlearning

Unlike standard GRPO, which samples all trajectories from a fixed policy, EEPO splits the rollout process into two stages separated by a lightweight unlearning step:

1. Stage 1 Sampling: The rollout model ($\pi_{rollout}$) generates the first half of the trajectories ($G/2$).
2. Unlearning Step: The model undergoes a temporary, single-step gradient update to "forget" the responses just sampled. This suppresses the probability of the dominant modes identified in Stage 1.
3. Stage 2 Sampling: The updated model generates the remaining trajectories ($G/2$). Because the dominant modes were suppressed, the model is forced to explore alternative, lower-probability regions of the output space.
4. Policy Update: All $G$ trajectories are used to compute rewards and advantages, and the main policy is updated using the standard GRPO objective. The unlearning effect is temporary and confined to the rollout model for that iteration; it does not alter the main policy parameters permanently.

Key Design Choices for Adaptive Unlearning

To make unlearning effective for RL exploration, three specific design choices are implemented:

• Entropy-Conditioned Gating: Unlearning is only triggered when the policy entropy drops below a threshold ($\alpha$). This ensures intervention happens only when exploration deteriorates (mode collapse onset), avoiding disruption during healthy exploration phases.
• Complementary Unlearning Loss: Instead of standard Negative Log-Likelihood (which penalizes low-probability tokens), EEPO uses a complementary loss: $L = -\log(1 - p)$. This imposes stronger penalties on high-probability (dominant) tokens, effectively pushing the model away from the modes it just sampled.
• Lightweight & Temporary: The unlearning is a single gradient step on the rollout model synchronized from the actor. It is decoupled from the main policy optimization, ensuring computational efficiency and preventing the unlearning from degrading the policy's overall capabilities.

3. Key Contributions

1. Novel Mechanism: Introduction of the "Sample-Then-Forget" paradigm, which actively disrupts the self-reinforcing loop of dominant modes during the rollout phase rather than just modifying the objective function.
2. Targeted Exploration: Unlike indiscriminate noise injection (e.g., higher temperature), EEPO strategically forces the model to explore under-sampled regions by temporarily suppressing visited trajectories.
3. Decoupling: Successfully decouples the exploration mechanism (unlearning on rollout model) from the exploitation mechanism (policy optimization), allowing for robust exploration without sacrificing training stability.
4. Comprehensive Evaluation: Extensive experiments across three different model families (Qwen2.5-3B, Llama3.2-3B-Instruct, Qwen3-8B-Base) and five challenging mathematical reasoning benchmarks.

4. Experimental Results

EEPO was evaluated on benchmarks including Minerva Math, OlympiadBench, AMC 2023, AIME 2024, and AIME 2025.

• Performance Gains: EEPO consistently outperformed standard GRPO and other exploration-enhanced baselines (e.g., higher temperature, increased entropy weight, DAPO clip-higher).
  • Qwen2.5-3B: +24.3% relative improvement in average accuracy.
  • Llama3.2-3B-Instruct: +33.0% relative improvement.
  • Qwen3-8B-Base: +10.4% relative improvement.
• Generalization: EEPO showed significant improvements on OOD benchmarks (e.g., AMC 2023), where GRPO and baselines often plateaued or declined due to overfitting.
• Entropy Dynamics: Analysis showed that while GRPO suffers from monotonic entropy collapse, EEPO maintains higher entropy throughout training. Notably, Stage 2 of EEPO exhibited higher entropy than Stage 1, confirming the mechanism successfully forces exploration of new regions.
• Efficiency: EEPO achieved these gains with negligible computational overhead, maintaining training times comparable to standard GRPO. In contrast, baselines like "Higher Temperature" significantly increased training time (by ~30%) due to longer, inefficient reasoning paths.

5. Significance

The paper provides a robust solution to the exploration-exploitation dilemma in RLVR for LLMs.

• Paradigm Shift: It moves beyond modifying loss functions or sampling parameters to actively manipulating the rollout dynamics via temporary unlearning.
• Practicality: The method is model-agnostic, computationally efficient, and easy to integrate into existing GRPO pipelines.
• Impact: By preventing entropy collapse, EEPO enables LLMs to discover diverse and superior reasoning strategies, leading to better generalization on complex, unseen mathematical problems. This is a significant step toward more reliable and capable reasoning models.