Adaptive Loss Balancing for Noise-Robust GRPO in Generative Recommendation

Technical Summary: Adaptive Loss Balancing for Noise-Robust GRPO in Generative Recommendation

1. Problem Statement

Generative Recommendation (GR) reformulates item retrieval as an autoregressive decoding task over learned Semantic IDs (SIDs). While GR offers superior coverage for long-tail and semantically related items compared to traditional embedding-based recall, it often lacks the fine-grained discrimination of ranking models. To bridge this gap, Reinforcement Learning (RL), specifically Group Relative Policy Optimization (GRPO), has been proposed to fine-tune GR policies using production rankers as reward models (RMs).

However, the paper identifies a critical flaw in applying RL uniformly to generative recommendation: reward model noise and exposure bias. Production rankers are trained on exposure-biased logs, making them unreliable for items outside their training distribution (e.g., new or long-tail items). The authors identify two failure modes of uniform reward application:

1. Easy Samples: When the policy already ranks the ground-truth item highly, reward scores cluster tightly. The resulting GRPO advantage collapses to near zero, leaving the gradient dominated by RM noise, which leads to over-optimization (reward hacking).
2. Hard Samples (RM-Unreliable): When the ground truth is a long-tail item, the RM may be locally miscalibrated, scoring popular distractors higher than the true target. Applying RL here pushes the policy away from the correct answer.

Stratified analysis reveals that while the aggregate influence of the RM across all samples is near zero, it provides substantial positive guidance conditionally—specifically on samples where the policy is uncertain (difficulty) and the RM exhibits high discriminability (reliability).

2. Methodology: AdaGRPO

To address these issues, the authors propose AdaGRPO, a framework that treats reward-guided optimization as selective admission rather than uniform pressure. The core innovation is a sample-level binary clip that gates the GRPO loss term based on two rank-based diagnostics computed from the rollout group.

The objective function for each instance $i$ is:
$$L_i(\theta) = L^{(i)}_{NLL}(\theta) + \lambda \alpha_i L^{(i)}_{GRPO}(\theta)$$
Where:

• $L_{NLL}$ is the supervised negative log-likelihood, which anchors all training instances to ensure stability.
• $\lambda$ is a global scaling coefficient.
• $\alpha_i \in \{0, 1\}$ is the clip coefficient determined by two diagnostics:

Diagnostic 1: Policy-Side Difficulty ($f_1$)
This condition checks if the ground-truth item ($y^*$) falls outside the top $\tau$-fraction of the policy's own ranking (based on log-likelihood).

• Logic: If $y^*$ is already in the top ranks, the supervised loss is sufficient, and the GRPO term is redundant (clipped). If $y^*$ is low-ranked, the policy is uncertain, and RL guidance is potentially needed.

Diagnostic 2: Reward-Side Discriminability ($f_2$)
This condition checks if the RM can reliably separate the ground truth from contextually irrelevant negatives.

• Logic: The RM must rank $y^*$ in the top $\tau$-fraction of a diagnostic pool (including rollouts and sampled negatives) AND rank all sampled negatives in the bottom $(1-\rho)$-fraction. If the RM fails to separate relevant from irrelevant items, the signal is deemed unreliable (clipped).

The Clip:
The final coefficient is $\alpha_i = f_1 \cdot f_2$. The GRPO loss is admitted only if the instance is both difficult for the current policy and supported by a locally reliable reward signal. Otherwise, the update reverts to pure supervision. This extends the PPO clipping principle from the ratio domain (bounding token updates) to the sample domain (excluding entire instances).

3. Key Contributions

• Stratified Analysis of Reward Reliability: The paper empirically demonstrates that the aggregate influence of production rankers on generative recommendation is negligible due to conflicting failure modes (noise on easy samples, bias on hard samples). It establishes that RM utility is conditional, not global.
• AdaGRPO Framework: A novel training objective that dynamically switches between supervised learning and RL based on per-sample diagnostics. It avoids the need for a learned value network or complex reward modeling, relying instead on rank-based thresholds derived from existing rollout statistics.
• Noise-Robust Optimization: By clipping the GRPO term on samples where the reward signal is either redundant or misleading, the method prevents the "collapse" of retrieval performance and hallucination often seen in standard GRPO training.

4. Experimental Results

The method was validated on a large-scale e-commerce dataset (175K sequences) and production A/B tests.

Offline Results:

• Performance: At the best intermediate checkpoint, AdaGRPO improved HR@10 from 11.01% (Base) to 12.18%, while keeping the hallucination rate $\le 0.22\%$.
• Stability: Unlike standard GRPO, which suffered a severe collapse in HR@10 (dropping to 10.39%) and a spike in hallucinations (0.89%) at the final checkpoint, AdaGRPO maintained robustness (HR@10: 11.63%, Hallucination: 0.27%).
• Comparison: AdaGRPO outperformed fixed NLL-GRPO mixtures across the retrieval-validity frontier. The addition of the reward-discriminability condition ($f_2$) further reduced hallucination and improved HR compared to using difficulty ($f_1$) alone.
• Stratified Analysis: Gains were most pronounced in intermediate-difficulty bins, confirming the method targets the specific regime where RL is beneficial.

Online A/B Tests:

• In production tests, AdaGRPO achieved statistically significant gains compared to the control:
  • Effective Item Page Views (IPV): +0.43%
  • User Click-Through Rate (UCTR): +0.27%
  • Dwell Time: +0.23%
• In contrast, a baseline using a fixed NLL-GRPO mixture showed no statistically significant gains in engagement metrics.

5. Significance and Claims

The paper concludes that the central challenge in applying RL to generative recommendation is not designing stronger rewards, but discerning when the reward signal can be trusted.

• Paradigm Shift: The work shifts the focus from global reward optimization to local, instance-wise trust. It argues that uniform application of RL signals in recommendation is inherently risky due to the mismatch between policy uncertainty and reward model calibration.
• Practical Utility: The results suggest that selective admission of policy-gradient updates allows generative recommenders to leverage noisy production rankers effectively without sacrificing the structural knowledge gained from supervised pre-training.
• Modesty: The authors acknowledge limitations, including the reliance on ground-truth items during training, sensitivity to hyperparameters ($\tau, \rho$), and the need for further validation across diverse user segments and catalogue freshness regimes. They frame the online results as strong production support for the method's robustness rather than a definitive causal proof of superiority over all possible baselines.