Policy Split: Incentivizing Dual-Mode Exploration in LLM Reinforcement with Dual-Mode Entropy Regularization

1. Problem Statement

Reinforcement Learning (RL) has been highly successful in training Large Reasoning Models (LRMs), leading to the emergence of models like o1 and R1. However, a fundamental conflict exists between exploration (diversity of thought) and exploitation (task accuracy) in entropy-guided RL for LLMs.

• The Trade-off: Traditional entropy regularization methods attempt to maximize entropy to encourage exploration. However, recent studies indicate that forcing high entropy often degrades accuracy, causing outputs to become incoherent or "degenerate."
• The Limitation: Existing approaches struggle to maintain the high accuracy of established LRMs while simultaneously incentivizing the diverse exploration needed for creative tasks or discovering novel reasoning paths. Simply adding a system prompt to request "high entropy" is insufficient to induce behavioral changes in the model's policy.

2. Methodology: Policy Split

The authors propose Policy Split, a novel RL paradigm that bifurcates a single model's policy into two distinct modes: a Normal Mode (optimized for accuracy) and a High-Entropy Mode (optimized for exploration). These modes share model parameters but are trained collaboratively with distinct objectives.

Core Components:

1. Dual-Mode Bifurcation via Prompting:

   • Normal Mode: Operates without a special system prompt, focusing on standard task correctness.
   • High-Entropy Mode: Activated by a specific natural language system prompt instructing the model to "explore unique thought processes" and "maximize diversity."
   • Note: The paper emphasizes that prompting alone is insufficient; explicit training is required to establish a robust high-entropy mode.

2. Dual-Mode Entropy Regularization:
   The framework employs a collaborative loss function where the two modes learn from each other's experiences (rollout sharing).

   • Normal Mode Loss: Optimized using standard GRPO-like advantages ($A_{i,t}$) focused solely on task correctness.
   • High-Entropy Mode Loss: Optimized with a modified advantage function ($\tilde{A}_{i,t}$) that includes:
     • Entropy Term: Encourages high-entropy rollouts.
     • KL Divergence Term: Encourages the high-entropy mode to diverge from the normal mode's behavior, preventing mode collapse.
     • Clamping: A technique to prevent the entropy signal from flipping the sign of the reward, ensuring training stability.
   • Rollout Sharing: The model samples rollouts from both modes. The normal mode learns from correct answers discovered by the high-entropy mode, while the high-entropy mode learns from the stable, correct responses of the normal mode to prevent it from collapsing into meaningless noise.

3. Objective Function:
   The total objective $J_\theta$ combines the PPO-like losses for both modes, weighted by a coefficient $\eta$ (entropy weight) and $\alpha$ (entropy computation weight), allowing the model to balance exploration and exploitation dynamically.

3. Key Contributions

• Novel Paradigm: Introduction of Policy Split, which decouples the pressure for accuracy from the need for exploration by maintaining two distinct policy behaviors within a single model.
• Collaborative Training: A mechanism where the normal and high-entropy modes mutually benefit. The high-entropy mode discovers novel solutions, while the normal mode provides a stability anchor.
• Dual-Mode Entropy Regularization: A specific mathematical formulation that uses KL divergence and entropy terms to ensure the high-entropy mode explores distinct behavioral patterns rather than just random noise.
• Prompt Generalizability: Demonstration that the high-entropy behavior learned during training can be triggered by unseen, rephrased prompts, indicating a learned capability rather than mere memorization of a specific string.

4. Experimental Results

The authors evaluated Policy Split on Qwen3 models (1.7B, 4B, 8B) across general reasoning tasks and creative writing.

• General Tasks (Math, Science, QA):

  • Policy Split consistently outperformed strong baselines (including GRPO, Entropy-Driven Advantage, and Entropy Regularization) in average accuracy across all model sizes.
  • It successfully "revived" the entropy of low-entropy LRMs without sacrificing accuracy.
  • The high-entropy mode, when triggered by the system prompt, achieved significantly higher entropy scores (diversity) compared to baselines, though with a slight trade-off in average accuracy for that specific mode.

• Creative Tasks:

  • In creative writing evaluations (using N-gram diversity, self-BLEU, and LLM-as-a-Judge metrics), the high-entropy mode of Policy Split significantly outperformed the base model and other RL baselines.
  • It achieved the highest scores in Creativity, Originality, and Imagery, proving that the method effectively unlocks unique, diverse responses.

• Ablation Studies:

  • Coefficient $\eta$: An optimal balance was found at $\eta=0.03$; too high caused instability, too low failed to encourage exploration.
  • Rollout Sharing: Essential for performance; without it, the high-entropy mode tended to collapse.
  • Mode Split: Analysis showed Policy Split created a significantly larger KL divergence between the two modes compared to vanilla GRPO, confirming distinct behavioral patterns.

5. Significance

• Solving the Accuracy-Diversity Trade-off: Policy Split demonstrates that it is possible to have both high accuracy and high diversity in LLMs by separating these objectives into collaborative modes rather than forcing a single policy to optimize for both simultaneously.
• Inference-Time Flexibility: The trained model offers a unique "dual-mode" capability. Users can switch between a "precision" mode (normal) and a "creative/exploratory" mode (high-entropy) simply by changing the system prompt, without needing separate models.
• Future Direction: The work suggests that future RL frameworks for LLMs should move beyond single-policy optimization toward multi-policy or collaborative modeling to achieve greater steerability and flexibility.

In conclusion, Policy Split provides a robust framework for enhancing LLM reasoning and creativity by structurally separating exploration and exploitation goals, achieving state-of-the-art results in both accuracy and diversity metrics.