Technical Summary: f-GRPO & Beyond: Divergence-Based Reinforcement Learning Algorithms for General LLM Alignment

1. Problem Statement

Large Language Model (LLM) alignment typically occurs in two distinct regimes:

1. Reinforcement Learning with Verifiable Rewards (RLVR): Feedback is provided via scalar rewards $r(x, y)$ (e.g., correctness in math or code). Here, aligned and unaligned responses are not explicitly observed as pairs; instead, they must be inferred from the reward signal. Standard approaches like Group Relative Policy Optimization (GRPO) use on-policy sampling and standardized advantages.
2. Preference Alignment: Feedback is provided via human preferences (chosen vs. rejected responses). Methods like Direct Preference Optimization (DPO) or f-divergence Optimizers (FDO) treat this as separating an aligned distribution $D_+$ from an unaligned distribution $D_-$.

Recent work (Haldar et al., 2025) established that preference alignment can be interpreted as estimating statistical divergences between $D_+$ and $D_-$. However, this divergence-based perspective has been limited to preference data. In RLVR, where explicit $D_+$ and $D_-$ distributions are unavailable, it remains an open question whether divergence estimation can provide a unified theoretical foundation. Furthermore, on-policy safety alignment often suffers from "reward hacking" when using learned reward models, as the policy may exploit model weaknesses rather than learning true safety constraints.

2. Methodology

The authors propose extending the divergence-based framework to general LLM alignment by inducing aligned and unaligned distributions directly from scalar rewards.

2.1 Inducing Distributions from Rewards

In RLVR, for a given prompt $x$, the authors define reward-aligned ($D_{(r,\theta)}^+$) and reward-unaligned ($D_{(r,\theta)}^-$) distributions based on the current policy $\pi_\theta$:

• $D_{(r,\theta)}^+$: Places mass on responses with rewards above the average reward $\mu_\theta^r(x)$.
• $D_{(r,\theta)}^-$: Places mass on responses with rewards below the average.

This allows the formulation of an on-policy divergence estimation problem where the goal is to separate high-reward responses from low-reward responses without explicit preference pairs.

2.2 f-GRPO (f-Group Relative Policy Optimization)

The authors derive f-GRPO, a class of on-policy objectives based on variational representations of $f$-divergences.

• Mechanism: Instead of reweighting samples solely by standardized advantages (as in GRPO), f-GRPO estimates the divergence between the induced reward-aligned and reward-unaligned distributions.
• Importance Sampling: The objective uses a group of $G$ sampled responses. It constructs importance weights using a softmax over rewards (corrected for the sampling policy likelihood) and truncates the update to only include above-average responses for the aligned term and below-average responses for the unaligned term.
• Loss Function: The loss combines the implicit policy reward $r_\theta$ with these reward-induced weights, effectively optimizing a variational $f$-divergence between the two induced distributions.

2.3 f-HAL (f-Hybrid Alignment Loss)

To address the limitations of pure on-policy methods in safety alignment (where rewards are learned and imperfect), the authors introduce f-HAL.

• Hybrid Objective: f-HAL interpolates between on-policy reward optimization (f-GRPO) and off-policy preference supervision (FDO).
• Formula: $L_{f-HAL} = \lambda L_{FDO} + (1-\lambda) L_{f-GRPO}$.
• Function: It uses on-policy rewards for exploration while anchoring the policy with off-policy preference data to prevent reward hacking. Theoretically, this estimates a divergence between mixtures of reward-induced and preference-induced distributions.

3. Key Theoretical Contributions

The paper provides rigorous theoretical guarantees for the proposed objectives:

• Divergence Estimation: The authors prove that f-GRPO and f-HAL are not merely heuristics but consistently estimate $f$-divergences between well-defined aligned and unaligned distributions (Theorem 4.3).
• Alignment Consistency: The fixed-point policy updates are shown to be alignment-consistent. Specifically, responses with higher density in the aligned distribution relative to the unaligned distribution are upweighted.
• Reward Improvement: Under canonical link functions (where $g^{-1}(f'_\infty) = \infty$), the authors prove monotone expected reward improvement. Unlike standard GRPO, which may assign non-zero probability to below-average responses, f-GRPO with canonical links concentrates probability mass strictly on above-average responses in the limit, guaranteeing convergence to the maximal-reward policy.
• Comparison to GRPO: The paper characterizes the fixed point of unclipped GRPO, showing it reweights the reference policy by the exponential of standardized rewards. However, f-GRPO offers a "sharper" separation, assigning zero mass to below-average responses under specific conditions, leading to stronger theoretical improvement guarantees.

4. Experimental Results

The authors evaluated f-GRPO and f-HAL on two primary tasks:

4.1 Math Reasoning (RLVR Task)

• Setup: Fine-tuning Qwen2.5-Math models (1.5B and 7B) on GSM8K, LIMR, and OpenRS datasets using verifiable rewards.
• Findings: f-GRPO consistently outperforms standard GRPO across various $f$-divergence choices (e.g., Pearson, KL, Hellinger).
  • On the challenging LIMR dataset, f-GRPO variants achieved higher Pass@1 scores and better aggregate rankings compared to GRPO.
  • The strongest gains were observed on more difficult datasets (LIMR, OpenRS), validating the theoretical advantage of sharper distribution separation.

4.2 Safety Alignment (Preference Task with Learned Rewards)

• Setup: Aligning models on a compliance-refusal dataset using a learned reward model (DeBERTa-v3-large) for on-policy methods, while comparing against off-policy direct aligners (DPO, KTO, BCO).
• Findings:
  • Reward Hacking: Pure on-policy methods (GRPO) performed poorly, exhibiting "blanket refusal" behavior (refusing safe prompts) due to exploiting the learned reward model.
  • Hybrid Mitigation: The hybrid f-HAL (with $\lambda=0.5$) successfully mitigated reward hacking. By combining on-policy exploration with off-policy preference supervision, it maintained context-awareness (answering safe prompts correctly) while refusing harmful ones.
  • Performance: Hybrid f-HAL often outperformed pure direct alignment on larger models (7B), suggesting that on-policy exploration is beneficial when model capacity is sufficient. On smaller models (1.5B), direct aligners remained slightly stronger, but f-HAL still significantly outperformed pure on-policy RL.

5. Significance and Claims

The paper claims to provide a unified and practical foundation for LLM alignment algorithms:

• Unification: It extends the divergence-based view of alignment from preference data to RLVR, showing that both regimes can be understood as estimating divergences between aligned and unaligned distributions.
• Theoretical Rigor: It establishes that these objectives offer stronger reward-improvement guarantees than standard on-policy methods like GRPO by explicitly modeling the separation of distributions.
• Practical Utility: The hybrid approach (f-HAL) offers a robust solution for safety alignment when verifiable rewards are unavailable, effectively balancing exploration (via rewards) and stability (via preferences) to prevent reward hacking.

The authors modestly note that while the theory supports monotone improvement for all canonical $f$-divergences, different divergences may induce different finite-sample dynamics. The experiments are limited to models up to 7B parameters to validate the framework's feasibility, with open-source code provided for reproducibility.