Technical Summary: LamPO – A Lambda-Style Policy Optimization for Reasoning Language Models

Problem Statement

Reinforcement learning with verifiable rewards (RLVR) has emerged as a dominant paradigm for enhancing reasoning capabilities in Large Language Models (LLMs) on tasks such as mathematics, coding, and scientific question answering. While Group Relative Policy Optimization (GRPO) has become a standard, critic-free alternative to PPO by normalizing rewards within a sampled group, it suffers from a critical limitation: information compression.

GRPO summarizes each group of sampled responses into scalar statistics (mean and standard deviation) to compute advantages. This approach discards fine-grained relational information among candidate responses. In reasoning tasks, where solutions may differ only subtly in quality (e.g., partially correct vs. near-correct), a scalar baseline can only indicate if a response is above or below the group average. It fails to explicitly preserve which specific responses a candidate outperforms and by how much. This limitation is particularly severe under sparse binary rewards, where many incorrect trajectories receive identical feedback, leading to weak credit assignment.

Methodology

The authors propose LamPO (Lambda-Style Policy Optimization), a method designed to retain the efficiency of critic-free optimization while recovering the lost relational structure within response groups.

1. Pairwise Decomposed Advantage (PDA)

Instead of computing a scalar advantage based on group statistics, LamPO replaces the group-relative advantage with a Pairwise Decomposed Advantage. For a response $o_i$ within a group of size $G$, the advantage $A_\lambda(o_i)$ is calculated as the average of pairwise reward gaps weighted by a confidence factor:

$$A_\lambda(o_i) = \frac{1}{G - 1} \sum_{j \neq i} (R(o_i) - R(o_j)) \sigma\left(\frac{\Delta s_{ij}}{\tau}\right)$$

Where:

• $R(o_i) - R(o_j)$ represents the pairwise reward gap, providing the learning direction.
• $\Delta s_{ij} = \log \pi_{\theta_{old}}(o_i | q) - \log \pi_{\theta_{old}}(o_j | q)$ is the difference in sequence log-probabilities under the old policy.
• $\sigma(\cdot)$ is the logistic sigmoid function acting as a confidence-aware weight.
• $\tau$ is a temperature parameter controlling the sensitivity of the weighting to log-probability differences.

This formulation aggregates pairwise comparisons, preserving intra-group ranking information. The weighting ensures that comparisons where the policy is more confident (larger log-probability difference) contribute more significantly to the update.

2. Policy Objective

LamPO retains the PPO-style clipped surrogate objective and the critic-free structure of GRPO. The standard advantage $\hat{A}_i$ is simply replaced with the computed PDA $A_\lambda(o_i)$ in the clipped objective function. The full objective includes a KL regularization term against a reference policy $\pi_{ref}$.

3. Dense Reference-Overlap Reward

To address reward sparsity when reference solutions are available, LamPO introduces a lightweight auxiliary reward based on ROUGE-L.

• The final reward is a linear combination: $R(o_i) = R_{ext}(o_i) + \lambda_{sem} R_{sem}(o_i)$.
• $R_{ext}$ is the sparse verifiable correctness reward (0 or 1).
• $R_{sem}$ is the ROUGE-L F1 score between the generated trace and the reference solution.
• $\lambda_{sem}$ controls the strength of this dense signal.

This auxiliary reward provides sequence-aware overlap signals, complementing the sparse correctness signal without replacing it.

Key Contributions

1. Identification of Limitation: The paper identifies the loss of intra-group relational information in scalar-baseline group-relative optimization as a key bottleneck in RLVR for reasoning tasks.
2. LamPO Algorithm: A novel, critic-free policy optimization method that utilizes Pairwise Decomposed Advantages with confidence-aware weighting to preserve ranking information within sampled groups.
3. Auxiliary Reward Integration: A simple, effective mechanism to reduce reward sparsity using ROUGE-L-based overlap when reference reasoning traces are available.
4. Empirical Validation: Demonstrated consistent improvements over GRPO and recent RLVR variants (DAPO, GSPO, SAPO) across multiple benchmarks and model families.

Experimental Results

The authors evaluated LamPO on four reasoning benchmarks: AIME24, AIME25, MATH-500, and GPQA-Diamond, using three model backbones: Qwen3-1.7B, Qwen3-4B, and Phi-4-mini.

• Performance: LamPO consistently outperformed GRPO and other baselines (CoT, SimpleRL-Zoo, PRIME, RLPR, DAPO, GSPO) across all model sizes.
  • On Qwen3-1.7B, LamPO achieved the highest average score (53.60), with notable gains on competition-style math tasks (AIME24: 46.67 vs. GRPO's 42.08).
  • On Qwen3-4B, LamPO improved the average score by 1.69 points over GRPO (77.09 vs. 75.40).
  • On Phi-4-mini, LamPO showed similar gains, indicating the method is not tied to a specific model family.
• Training Dynamics: Training curves revealed that LamPO produces smoother reward improvements and more stable completion lengths compared to GRPO. This stability is attributed to the structured signal provided by pairwise gaps, which reduces oscillatory updates caused by high-variance scalar normalization, especially in smaller models.
• Ablation Studies:
  • Temperature ($\tau$): A value of $\tau=1.5$ yielded the best results. Lower values ($\tau=0.5$) made the weighting too sensitive, while higher values ($\tau=5.0, 7.0$) smoothed the weighting too much, weakening pairwise discrimination.
  • Semantic Reward: Removing the dense auxiliary reward ($\lambda_{sem}=0.0$) reduced performance, confirming that reference-overlap shaping improves sample efficiency under sparse rewards.

Significance and Claims

The paper claims that LamPO offers a more effective credit assignment mechanism for reasoning tasks where solution quality varies subtly. By shifting from scalar group statistics to pairwise decomposed advantages, LamPO provides a richer learning signal that explicitly leverages the relative ordering of candidate responses.

The authors emphasize that LamPO maintains the computational efficiency and stability of the GRPO framework (being critic-free and using clipped updates) while addressing the specific weakness of information loss in group-relative methods. The integration of a lightweight ROUGE-L reward further enhances training efficiency when reference data is available, without compromising the reliance on verifiable correctness.

Limitations Acknowledged:

• Computational Cost: The pairwise comparison introduces an $O(G^2)$ complexity relative to group size $G$, though this is considered negligible for typical group sizes in RLVR.
• Reward Reliability: The method depends on reliable reward signals; noisy or subjective rewards could amplify errors in pairwise gaps.
• Auxiliary Reward Scope: The ROUGE-L reward captures lexical overlap rather than full reasoning correctness and requires reference solutions, which may not exist for all tasks.
• Generalization: Experiments are currently limited to mathematical and scientific reasoning; generalization to open-ended dialogue or multimodal tasks remains an open question.