🤖 machine learning

F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

The paper proposes F-GRPO, a difficulty-aware reinforcement learning method that mitigates the tendency of standard group-based algorithms to overfit common solutions and neglect rare correct trajectories by down-weighting high-success updates, thereby significantly improving mathematical reasoning performance across various baselines without increasing computational costs.

Original authors: Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov, Viacheslav Sinii, Alexey Malakhov, Daria Korotyshova, Daniil Gavrilov

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov, Viacheslav Sinii, Alexey Malakhov, Daria Korotyshova, Daniil Gavrilov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Group Study" Problem

Imagine you are teaching a student (an AI model) how to solve a difficult math problem. To help them learn, you don't just show them one answer; you ask them to generate eight different attempts (a "group") at the same time. You then look at these eight attempts, compare them, and tell the student: "Hey, most of your answers were wrong, but this one was right. Let's make you more likely to do that one next time."

This method is called Group-Relative Policy Optimization (GRPO). It's like a study group where the teacher only gives feedback based on what the group actually produced.

The Problem:
The paper argues that if your study group is too small, you might not see the right answer at all. But if your group is just the "right" size (not too small, not huge), a weird thing happens:

The group does find the right answer.
The teacher says, "Great job on that one!"
The student gets so excited about that one specific right answer that they stop trying to find other ways to solve the problem. They become obsessed with that single solution and forget about all the other valid ways to get there.

In the AI world, this is called "Distribution Sharpening." The AI gets really good at finding one common answer but loses the ability to find rare, creative, or difficult answers. It's like a student who memorizes the answer key for the most common test questions but fails completely when the teacher asks a tricky, unusual question.

The Core Discovery: The "Goldilocks" Trap

The authors did some math to prove that this "forgetting" happens most often when the group size is medium.

Tiny Groups (Size 2): The group often fails to find any right answer. The teacher says, "Nothing worked this time," so the student doesn't change their habits much. They stay safe and diverse, but they don't learn much.
Huge Groups (Size 128+): The group finds every possible right answer, including the rare ones. The teacher says, "Look, you found the common one AND the rare one!" The student learns everything. But this is too expensive to do on a computer (it costs too much money and time).
Medium Groups (Size 8-16): This is the trap. The group finds the common right answer (so the teacher gives feedback), but it misses the rare right answer. The student thinks, "The common answer is the only one that matters," and they stop exploring the rare ones.

The Analogy:
Imagine you are looking for a specific rare coin in a jar of 1,000 coins.

If you grab 2 coins, you probably won't find the rare one. You don't learn anything.
If you grab 500 coins, you will definitely find the rare one. You learn everything.
If you grab 10 coins, you might find the common coins but miss the rare one. You then conclude, "The rare coin doesn't exist," and you stop looking for it.

The Solution: F-GRPO (The "Difficulty-Aware" Coach)

The authors propose a fix called F-GRPO. They realized that when a group finds many correct answers, the AI gets too confident and starts ignoring the rare ones.

So, they added a "difficulty weight" inspired by a technique called Focal Loss.

How it works:

The Old Way: If the group finds 5 correct answers out of 8, the teacher gives a huge "High Five" and tells the AI to focus heavily on those answers.
The New Way (F-GRPO): The teacher looks at the group and says, "Wow, you found 5 correct answers! That's easy for you right now. I'm going to turn down the volume on this feedback."
- If the group finds few correct answers (it was a hard struggle), the teacher turns the volume up and says, "This was hard, pay close attention to what worked!"
- If the group finds many correct answers (it was easy), the teacher turns the volume down so the AI doesn't get too obsessed with the obvious solutions.

The Result:
By turning down the volume on "easy" groups, the AI is forced to keep exploring. It doesn't stop looking for the rare, difficult solutions just because it found an easy one.

What the Experiments Showed

The team tested this on several AI models (like Qwen and Llama) using math problems and logic puzzles.

The "Rare" Test: They checked how well the AI could find any correct answer if they gave it 256 tries (instead of just 1).
- Without the fix: As the AI got better at the easy answers, its ability to find the rare answers dropped.
- With F-GRPO: The AI kept its ability to find the rare answers high, even while getting better at the easy ones.
The "Maze" Test: They used a maze where there is only one correct path. Even in this simple case, the old method made the AI forget the path if it got lucky early on. F-GRPO kept the AI on track.
Efficiency: They achieved these results without increasing the group size. They didn't need to ask the AI to generate 100 answers; they just needed to change how they listened to the 8 answers it already generated.

Summary

The paper says: "Don't let your AI get too comfortable with the obvious answers."

When an AI learns from a group of attempts, it tends to forget the rare, difficult solutions if the group is a medium size. The authors fixed this by creating a "volume knob" (F-GRPO) that lowers the importance of easy, high-success groups. This forces the AI to keep exploring and ensures it doesn't lose its ability to solve the hard, rare problems.

Technical Summary: F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has become a standard paradigm for post-training Large Language Models (LLMs) on reasoning tasks. However, a growing body of evidence suggests that RLVR often sharpens the output distribution toward solutions already accessible to the base model, potentially degrading performance on metrics like pass@k (where k is large) by narrowing solution diversity.

Most modern RLVR systems, such as Group Relative Policy Optimization (GRPO), rely on group sampling (rolling out $N$ responses per prompt) to estimate advantages. While small groups ( $N=2$ ) are computationally efficient and large groups provide broad coverage, practical constraints often limit $N$ to intermediate sizes (e.g., $N=8$ ). The paper identifies a critical gap in understanding how these finite group sizes affect optimization dynamics. Specifically, it posits that intermediate group sizes may create a "tail-miss" regime where updates are active (mixed rewards) but fail to sample rare-correct trajectories, leading to a concentration of probability mass on common, sampled solutions while suppressing unsampled correct behaviors.

Methodology

Theoretical Analysis

The authors derive the probability of a tail-miss event ( $B_{E,N}(x)$ ), defined as an active update (where the group contains both correct and incorrect rollouts) that fails to sample a specific low-mass correct subset $E_x$ .

Non-Monotonic Behavior: The probability of this event is non-monotonic with respect to group size $N$ . Small $N$ often results in inactive updates (all correct or all incorrect), preserving diversity through inactivity. Large $N$ ensures coverage of rare modes. However, intermediate $N$ maximizes the probability of active updates that miss rare-correct regions.
Redistribution Mechanism: Using a categorical policy framework, the authors analyze how probability mass is redistributed. They show that even when the total correct mass ( $Q_{pos}$ ) increases, the unsampled-correct mass ( $Q_{u,pos}$ ) can decrease. This occurs because reward-positive batches push unsampled logits downward through baseline subtraction and softmax normalization, effectively "forgetting" correct solutions that were not sampled in the current group.

Proposed Solution: F-GRPO

To mitigate this without increasing computational cost (rollout size), the authors propose F-GRPO (Focal Group Relative Policy Optimization).

Focal Weighting: Inspired by Focal Loss, the method introduces a difficulty-aware scaling coefficient $g(x)$ applied to the group-relative advantage.
Mechanism: The weight is defined as $g(x) = (1 - \hat{\mu}_{pos}(x))^\gamma$ , where $\hat{\mu}_{pos}(x)$ is the empirical success rate of the sampled group (number of correct rollouts $X$ divided by $N$ ).
Effect: This down-weights updates from groups with high empirical success rates (where the model has already "learned the obvious" for that prompt) and preserves the signal from mixed-success groups where rare correct behaviors might still be underrepresented. It is a lightweight modification that scales the advantage signal uniformly for all rollouts in a prompt without requiring additional networks or changing reward values.

Key Contributions

Tail-Miss Probability Derivation: The paper explicitly characterizes the probability of active updates missing rare-correct subsets, demonstrating its non-monotonic dependence on group size. This helps reconcile conflicting prior findings regarding optimal group sizes.
Unsampled-Correct Mass Analysis: The authors show theoretically and empirically that total correct mass can grow while unsampled-correct mass shrinks, explaining the mechanism behind distribution sharpening in finite-group settings.
Single-Solution Validation: Using a Maze environment where each prompt has exactly one correct trajectory, the paper demonstrates that pass@K degradation can occur even without multiple correct solutions per prompt, isolating the effect to target-sequence coverage rather than intra-prompt diversity.
F-GRPO Algorithm: A simple, plug-in reweighting scheme applicable to GRPO, DAPO, and CISPO that improves pass@256 while maintaining or improving pass@1, without increasing rollout costs.

Experimental Results

The authors evaluated F-GRPO on Qwen2.5-7B, Qwen3-4B-Base, and Llama-3.2-3B-Instruct across various benchmarks (MATH500, AIME, AMC, GPQA, IFEval, SynLogic).

Categorical Simulation: Confirmed that intermediate group sizes lead to the lowest retained positive mass ( $M_{ret}$ ), while Focal weighting ( $\gamma=1$ ) significantly improves mass retention in this "concentration zone."
Maze Experiments: In a single-solution setting, F-GRPO reduced the early training-time drop in pass@K and improved final pass@1 and pass@256 across all tested group sizes ( $N \in \{4, \dots, 128\}$ ).
LLM Benchmarks (N=8):
- Math Performance: On Qwen2.5-7B, F-GRPO improved average math pass@256 from 64.1 (GRPO) to 70.3, and from 69.3 (DAPO) to 72.5.
- OOD Performance: Out-of-domain (OOD) pass@256 also improved consistently across all methods (e.g., GRPO: 55.9 $\to$ 63.3; DAPO: 58.4 $\to$ 63.6).
- Efficiency: F-GRPO at $N=8$ achieved performance comparable to standard GRPO at $N=32$ on pass@256, effectively recovering the benefits of larger groups with 4 $\times$ less compute.
- Generalization: The method transferred effectively across different optimizers (GRPO, DAPO, CISPO) and model scales.
Ablation: The gains were not replicated by simple learning rate reduction or standard regularization (entropy/KL), suggesting the improvement stems specifically from the success-based reweighting mechanism.

Significance and Claims

The paper claims to provide a theoretical and practical resolution to the "sharpening" problem in group-relative RLVR. By identifying that intermediate group sizes maximize the risk of missing rare-correct trajectories, the authors argue that simply increasing $N$ is not always feasible due to computational constraints.

F-GRPO offers a low-cost mitigation strategy that:

Recovers the pass@256 performance typically lost when using intermediate group sizes.
Preserves or improves pass@1 (single-sample accuracy).
Works across different RLVR variants and model architectures.
Does not require increasing the rollout budget or adding auxiliary networks.

The authors position F-GRPO as a necessary adjustment to the standard RLVR pipeline, ensuring that policy updates do not inadvertently reinforce only the "obvious" solutions exposed in a finite sample while forgetting the "rare" correct behaviors necessary for robust reasoning.