Are complicated loss functions necessary for teaching LLMs to reason?

1. Problem Statement

Recent advances in Large Language Models (LLMs) for mathematical reasoning have been driven by post-training techniques, particularly Group Relative Policy Optimization (GRPO). While GRPO has demonstrated state-of-the-art performance by eliminating the need for a separate value model (critic) and using group-relative advantage estimation, its loss function is complex. It combines three distinct components:

1. Group-relative advantage estimation.
2. PPO-style clipping (to constrain policy updates).
3. KL regularization (to prevent deviation from a reference model).

The authors question whether this complexity is strictly necessary. Specifically, they investigate if the PPO-style constraints (clipping and policy ratios) and the specific handling of negative feedback are essential for fostering reasoning behaviors, or if simpler Reinforcement Learning (RL) variants can achieve comparable or superior results with greater transparency and efficiency.

2. Methodology

The authors conducted a systematic ablation study on the GRPO loss function to isolate the necessity of its components. They evaluated several variants across three model families (Qwen2.5-0.5B, Qwen2.5-1.5B, and Llama3.2-1B) trained on the GSM8K dataset.

Key Variants Analyzed:

1. Positive-Only GRPO (GRPO-pos): Removes negative feedback by setting advantages to zero if they are below the baseline. This tests if training solely on "good" actions is sufficient.
2. RGRA (REINFORCE with Group Relative Advantage): The authors' proposed method. It retains the group-relative advantage estimation but removes PPO-style clipping and policy ratio terms. It uses a standard REINFORCE gradient update with the group-relative advantage.
   • Formula: $\nabla_\theta J_{RGRA} = \mathbb{E} [\sum \nabla_\theta \log \pi_\theta \cdot \hat{A}_{i,t} - \beta \nabla_\theta D_{KL}]$
3. REINFORCE with Direct Rewards: Removes group-relative advantage estimation entirely, training directly on raw reward signals.
4. RAFT (Rejection Sampling Fine-Tuning): A simpler baseline that samples multiple responses, ranks them, and performs supervised fine-tuning (SFT) on the top-ranked responses using cross-entropy loss.

Experimental Setup:

• Training Data: 1,800 instances from GSM8K (decontaminated).
• Benchmarks: Nine benchmarks covering English Math (GSM8K, MATH, Gaokao2023, Olympiad, AMC23), Chinese Math (CMATH, CN-Middle-School), and STEM (MMLU-STEM, Gaokao2024).
• Reward System: Binary correctness (1.0) and format compliance (0.1).
• Group Size: 8 completions per prompt.

3. Key Findings & Results

A. Necessity of Negative Feedback

Training exclusively on positive advantages (GRPO-pos) or using RAFT led to severe instability, particularly in smaller models (0.5B).

• Observation: Models collapsed into "reward hacking," generating minimal-length, degenerate outputs to maximize reward without reasoning.
• Conclusion: Negative feedback (penalizing actions below the baseline) is essential for stabilizing learning and preventing policy collapse.

B. Necessity of PPO-Style Clipping

The RGRA variant, which removes PPO clipping and policy ratios, outperformed standard GRPO in the majority of cases.

• Performance: RGRA achieved the highest average performance across English Math benchmarks, surpassing GRPO in 17 out of 27 individual comparisons.
• Stability: RGRA maintained stable training dynamics comparable to GRPO, proving that the clipping mechanism is not strictly necessary when initializing from strong policy models.
• Reasoning Emergence: Unlike RAFT or Positive-Only GRPO (which output direct answers), RGRA and GRPO successfully induced emergent reasoning behaviors, such as explicit re-evaluation of intermediate steps.

C. Importance of Advantage Estimation

The REINFORCE with Direct Rewards variant (using raw rewards without advantage estimation) caused training collapse even in larger models (1.5B).

• Conclusion: Advantage estimation (normalizing rewards relative to a group) is crucial for stability, even more so than PPO clipping.

D. Quantitative Results

• Qwen2.5-1.5B: RGRA achieved an average score of 38.3 on English Math benchmarks vs. GRPO's 37.3.
• Chinese Math: RGRA achieved 69.3 average accuracy vs. GRPO's 65.7.
• STEM: RGRA showed consistent improvements over GRPO on Qwen models.

4. Key Contributions

1. Simplification of GRPO: The paper demonstrates that the complex PPO-style constraints in GRPO are redundant for mathematical reasoning tasks.
2. Proposal of RGRA: Introduction of a simplified, REINFORCE-based algorithm that retains group-relative advantage estimation but discards clipping, offering a more transparent and efficient alternative.
3. Identification of Critical Components:
   • Essential: Group-relative advantage estimation and negative feedback.
   • Non-Essential: PPO-style clipping and policy ratio constraints.
4. Empirical Superiority: RGRA not only matches but often exceeds the performance of the more complex GRPO across diverse benchmarks and model sizes.

5. Significance

This work challenges the prevailing assumption that complex RL algorithms (like PPO/GRPO with clipping) are required for aligning LLMs on reasoning tasks.

• Efficiency: By removing the clipping mechanism and policy ratio calculations, RGRA simplifies the training pipeline, potentially reducing computational overhead and implementation complexity.
• Transparency: The results suggest that the "magic" of GRPO's success lies primarily in the group-relative advantage estimation and the inclusion of negative feedback, rather than the specific PPO constraints designed for general RL stability.
• Future Direction: The findings pave the way for designing more streamlined post-training strategies for reasoning-focused LLMs, potentially extending to larger models and non-mathematical domains.

In summary, the authors conclude that simpler REINFORCE-based approaches (specifically RGRA) are sufficient and often superior for teaching LLMs to reason, provided that group-relative advantage estimation and negative feedback are preserved.