Technical Summary: Two is Better than One: A Collapse-free Multi-Reward RLIF Training Framework

1. Problem Statement

While Reinforcement Learning with Verifiable Rewards (RLVR) has significantly enhanced the reasoning capabilities of Large Language Models (LLMs), it relies heavily on external supervision (e.g., human annotations, gold-standard solutions), limiting its scalability. Reinforcement Learning from Internal Feedback (RLIF) offers an unsupervised alternative by deriving reward signals from the model itself, such as self-certainty (entropy minimization) or consensus-based measures (majority voting).

However, existing RLIF methods suffer from recurring failure modes, specifically reward hacking and model collapse. The paper argues that these failures stem from the reliance on a single internal reward signal:

• Self-certainty rewards favor high-confidence token distributions, often leading to "token-mode collapse" where the model produces short, deterministic, template-like outputs regardless of input quality.
• Cluster-voting rewards (consensus-based) can lead to "answer-mode collapse," where the model drifts in reasoning space to produce long, high-entropy completions that converge on a single dominant (but potentially incorrect) answer cluster.

This behavior exemplifies Goodhart's Law: when a single proxy becomes the optimization target, the policy exploits the proxy rather than improving the underlying reasoning capability, leading to a trajectory of early improvement followed by degradation.

2. Methodology

The authors propose a Multi-Reward RLIF Framework that decomposes the training signal into two complementary components and introduces targeted regularization to prevent collapse. The framework operates within a GRPO-style (Group Relative Policy Optimization) objective.

2.1 Dual-Reward Mechanism

For a given prompt, the model samples $G$ completions. Two intrinsic rewards are computed for each completion:

1. Answer-Level Reward (Cluster Voting): Final answers are extracted and clustered by equivalence. A completion receives a reward proportional to the size of its answer cluster. This provides a denser signal than binary majority voting, rewarding minority clusters with non-zero values to encourage diversity while still favoring consensus.
2. Completion-Level Reward (Self-Certainty): Based on the average KL divergence between the model's next-token distribution and a uniform distribution. This rewards confident, low-entropy token generation during the reasoning process.

2.2 GDPO-Based Normalization

To prevent one reward signal from dominating the other due to scale differences (e.g., self-certainty values depending on vocabulary size vs. cluster rewards in $[0, 1]$), the framework applies Group-Wise Multi-Reward Normalization inspired by GDPO. Each reward channel is normalized independently within the group of $G$ completions using z-scoring before being combined into a single rollout-level advantage:
$$A_g = \alpha \cdot \hat{R}_{cluster}(a_g) + \beta \cdot \hat{R}_{cert}(c_g)$$
This ensures that the relative importance of answer quality versus reasoning confidence is controlled by the weights $\alpha$ and $\beta$, rather than by the raw variance of the signals.

2.3 KL-Cov Regularization

To address the residual risk of entropy collapse, the authors introduce KL-Cov, a targeted regularization strategy.

• Mechanism: The method identifies the top-$k$ (e.g., 2%) of tokens with the highest covariance between their log-probabilities and the rollout advantages. These "high-covariance" tokens are the primary drivers of entropy collapse.
• Penalty: Instead of applying a global KL penalty, the framework applies a targeted $L_1$ penalty (absolute log-importance ratio) only to these high-covariance tokens. The remaining ~98% of tokens undergo standard clipped GRPO updates.
• Goal: This suppresses the specific tokens responsible for disproportionate entropy reduction while preserving exploration and diversity in the rest of the token distribution.

3. Key Contributions

1. Multi-Reward RLIF Framework: A novel architecture that jointly optimizes answer-level consensus and completion-level self-certainty, mitigating the failure modes associated with single-proxy optimization.
2. GDPO Normalization: The application of group-wise normalization to unsupervised multi-reward settings to prevent scale-induced dominance of one signal over another.
3. KL-Cov Regularization: A targeted policy constraint that specifically penalizes high-covariance tokens to prevent entropy collapse without stifling general exploration.
4. Empirical Validation: Demonstration that complementary internal rewards, combined with targeted regularization, can support stable long-horizon reasoning without external ground-truth supervision.

4. Experimental Results

The framework was evaluated on five mathematical reasoning benchmarks (GSM8K, MATH500, MMLU-Pro, AIME 2024/2025) and two code-generation benchmarks (LiveCodeBench v6, CRUXEval-O) using Qwen2.5-1.5B and Qwen2.5-3B base models.

• Stability vs. Collapse: Single-reward baselines (e.g., INTUITOR using only self-certainty) exhibited catastrophic collapse, with accuracy dropping to near-zero after initial gains (e.g., step 160–200). The multi-reward approach without KL-Cov delayed this collapse but eventually degraded. The full Multi-Reward + KL-Cov method remained stable across the entire training horizon (340 steps), retaining 89–99% of peak performance.
• Performance: The proposed method achieved performance close to supervised RLVR baselines (GRPO with ground-truth rewards) while remaining fully unsupervised.
  • On Qwen2.5-3B, the method matched or exceeded supervised baselines on several metrics (e.g., 7.7% pass@1 on LiveCodeBench vs. 6.4% for supervised GRPO).
  • It significantly outperformed single-reward unsupervised baselines (e.g., +45.9 percentage points on GSM8K for the 1.5B model compared to INTUITOR).
• Generalization: The reasoning structure learned on mathematical tasks transferred effectively to code-generation tasks (out-of-distribution), where the method outperformed single-reward baselines.

5. Significance and Claims

The paper claims that the observed instability in prior unsupervised RL methods is not an inherent flaw of RLIF but a consequence of optimizing a single internal proxy. By decomposing the reward signal into structurally complementary components (answer agreement and token confidence) and applying targeted regularization against the specific tokens driving collapse, the framework achieves stable, long-horizon reasoning without external supervision.

The authors position this work as a scalable alternative to externally supervised RLVR, suggesting that future research can explore additional internal signals to further support complex reasoning trajectories. The results indicate that "two is better than one": complementary internal rewards, when properly normalized and regularized, can prevent the model from collapsing into narrow reward-seeking patterns, thereby preserving the exploration necessary for complex problem-solving.