Technical Summary: Perturbed Parameter Policy Optimization (3PO) for RLVR

Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) is a prominent post-training paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs) without relying on human demonstrations. However, current RLVR algorithms, particularly Group Relative Policy Optimization (GRPO), face significant challenges related to the exploration-exploitation trade-off.

The primary limitation lies in action-space exploration. Standard methods control exploration by adjusting the softmax temperature during token generation. As illustrated in the paper, temperature scaling only alters the variance of the token distribution while preserving the relative ordering of tokens. This restricts the model to exploring trajectories within the same "shape" of the solution space, often failing to discover novel, high-reward paths that have low initial probability. Consequently, this leads to:

1. Premature Entropy Collapse: The policy becomes overly deterministic, generating homogeneous rollouts.
2. Zero-Advantage Groups: In GRPO, if all $G$ rollouts for a prompt receive the same reward (e.g., all correct or all incorrect), the advantage signal becomes zero, stalling learning.
3. Degeneration: High-temperature action-space exploration often leads to malformed or incorrect outputs (e.g., token repetition) rather than valid alternative solutions.

Existing solutions like entropy regularization or advantage shaping aim to stabilize training but do not fundamentally alter the generator's parameters to enable deeper exploration.

Methodology: Perturbed Parameter Policy Optimization (3PO)

The authors propose Perturbed Parameter Policy Optimization (3PO), a family of methods that shifts the exploration axis from the action space (tokens) to the parameter space (weights). Instead of merely reordering tokens, 3PO samples different policies from an approximate posterior distribution during the rollout generation phase.

Core Mechanism

The method utilizes Variational Learning, specifically the IVON (Iterative Variational Online Newton) optimizer, to learn a distribution over neural network parameters $\theta$.

• Posterior Sampling: At each training step, the model parameters are perturbed by adding learned noise: $\hat{\theta} = m + \sigma \odot z$, where $z \sim \mathcal{N}(0, I)$.
• Noise Control: The variance $\sigma$ is inversely proportional to the diagonal Hessian $h$ and a scaling factor $\lambda$ (Effective Sample Size, ESS). A smaller $\lambda$ increases noise, promoting exploration, while a larger $\lambda$ reduces noise, focusing on exploitation.
• Orthogonality: This approach is compatible with existing action-space techniques (e.g., temperature scaling, entropy regularization) as it operates on a different axis.

Three Variants of 3PO

The paper introduces three strategies for integrating parameter noise into the GRPO framework:

1. B3PO (Batched Noising):

   • Samples a single weight perturbation $\hat{\theta}$ per gradient step.
   • This single perturbed model generates all rollouts for the batch.
   • Limitation: Similar to standard GRPO, if the single sampled policy is poor, the entire batch may lack diversity.

2. M3PO (Multiple Monte-Carlo Samples):

   • Samples $M$ distinct weight perturbations from the posterior.
   • Generates rollouts for each perturbation and accumulates the GRPO loss across all $M$ samples before performing a single weight update.
   • Trade-off: To maintain equal computational cost, the group size $G$ is reduced per perturbation ($G/M$), which can limit the diversity within individual advantage groups.

3. C3PO (Chunked Noising):

   • Samples $N$ distinct weight perturbations per batch.
   • The total group of $G$ rollouts is partitioned across these $N$ models (e.g., $G/N$ rollouts per model).
   • Advantages are calculated over the full group of $G$ responses, which now includes trajectories from $N$ different policies.
   • Goal: Maximizes group diversity by ensuring the advantage calculation leverages multiple distinct parameter configurations, directly addressing the zero-advantage problem.

Key Contributions

1. Parameter-Space Exploration for LLMs: The paper establishes that perturbing model weights (rather than just token probabilities) is an effective mechanism for RLVR, allowing models to explore different regions of the solution space that action-space noise cannot reach.
2. The 3PO Framework: The introduction of B3PO, M3PO, and C3PO, with a specific focus on C3PO as the most effective strategy for maximizing group diversity without increasing computational overhead.
3. Empirical Validation: Comprehensive experiments on OLMo-3-1025-7B and Qwen2.5-Math-7B across mathematical reasoning (AIME, MATH-500, AMC) and code generation (LiveCodeBench) tasks.
4. Analysis of Zero-Advantage Groups: Demonstration that parameter-space exploration significantly reduces the frequency of zero-advantage groups and malformed rollouts compared to action-space baselines.

Experimental Results

• Performance Gains: 3PO variants consistently outperform standard GRPO and action-space baselines (Entropy Regularization, Polaris, KLCov).
  • C3PO achieved the best average performance across both model families.
  • Gains were most pronounced on harder benchmarks (e.g., AIME 2024–26, LiveCodeBench), where pre-RL models are less capable and more prone to generating invalid rollouts.
  • On Qwen2.5-Math, C3PO improved the average Pass@1 by approximately 1.5% over standard GRPO, with specific gains of up to 5.83% on AIME tasks.
• Exploration Quality:
  • Rescuing Dead Groups: 3PO methods, particularly C3PO and M3PO, consistently "rescued" more zero-advantage groups (producing at least one correct rollout where GRPO produced none) than action-space baselines.
  • Reduced Degeneration: Unlike temperature-based methods (Polaris) which often produce malformed outputs, and Entropy Regularization which produces incorrect but well-formed outputs, 3PO produced fewer malformed and incorrect rollouts while maintaining high diversity.
• Hyperparameter Sensitivity:
  • The ESS scaling factor $\lambda$ is critical. $\lambda = 10^9$ was found to be a robust default. $\lambda = 10^8$ (too much noise) caused training collapse, while $\lambda = 10^{10}$ (too little noise) plateaued performance.
  • C3PO benefits from $N \ge 2$ perturbations, with $N=4$ showing strong results.
  • M3PO benefits from removing the "equal-compute" constraint; when allowed to use full group sizes with multiple samples, performance improves significantly.

Significance and Claims

The paper posits that parameter-space exploration provides a necessary and complementary control lever for RLVR that action-space methods lack. By learning an approximate posterior over weights, the model can adaptively inject noise to explore promising alterations in token distributions that are structurally different from the current policy.

The authors claim that:

1. Diversity is Key: The primary driver of success is the increased diversity within GRPO groups, which prevents the stagnation caused by zero-advantage groups.
2. Efficiency: 3PO achieves these improvements at a near-identical FLOPs cost to standard GRPO (specifically C3PO), making it a practical drop-in replacement.
3. Generalizability: The method does not require a learned noise prior from a specific SFT phase; initializing the Hessian with a constant value allows the method to work effectively on off-the-shelf checkpoints.

The work concludes that while parameter-space exploration has a history in classical RL, its application to LLMs via variational learning (3PO) offers a robust solution to the exploration bottlenecks currently limiting RLVR performance, particularly on complex reasoning and coding tasks.