Label-Free Reinforcement Learning via Cross-Model Entropy

Technical Summary: Label-Free Reinforcement Learning via Cross-Model Entropy

Problem Statement
Post-training large language models (LLMs) via reinforcement learning (RL) is currently bottlenecked by the requirement for a reliable reward signal. Existing approaches face a dichotomy:

1. Ground-truth verifiers: Methods like PPO and GRPO rely on automatically checkable correctness (e.g., in mathematics or code execution). This restricts RL training to domains with objective answers, excluding open-ended instruction following.
2. Human preference labels: Methods like RLHF and DPO require expensive human annotation and are susceptible to reward hacking.
3. Label-free self-referential signals: Recent alternatives (e.g., TTRL, Evol-RL, RENT) use the generator's own outputs to derive rewards via majority voting or token entropy. However, these are structurally flawed for open-ended tasks: if a model is systematically wrong but internally consistent, self-referential signals reinforce errors rather than correcting them. Furthermore, majority voting requires a canonical answer, making it inapplicable to open-ended generation.

Methodology: Cross-Model Entropy (CME)
The authors propose Cross-Model Entropy (CME), a label-free reward signal designed to extend RL to open-ended instruction following without ground-truth labels or human preferences.

• Core Concept: Instead of asking a generator to evaluate its own outputs, CME measures how "surprising" a generator's response is to a separate, independent verifier model ($\pi_\phi$).
• Reward Definition: The reward is defined as the negative log-likelihood of the generator's response under the verifier model.
  $$r_{i,t} = -CME_{i,t} = \log \pi_\phi(y_{aligned, i, t} \mid x, y_{aligned, i, <t})$$
  Where $y_{aligned}$ represents the generator's tokens aligned to the verifier's token space.
• Theoretical Grounding: Maximizing expected CME is mathematically equivalent to minimizing the reverse KL divergence $D_{KL}(\pi_\theta \parallel \pi_\phi)$ (up to a generator entropy term). This effectively distills the generator's policy toward the verifier's distribution. If the verifier is more competent in-domain than the generator, this drives the generator toward higher-quality outputs.
• Implementation Details:
  • Integration: CME is integrated into the GRPO (Group Relative Policy Optimization) framework. It replaces the standard scalar reward with a token-level, continuous signal.
  • Token Alignment: To handle mismatches between generator and verifier tokenizers (e.g., Qwen vs. Gemma), the authors employ a character-level alignment procedure. Verifier log-probabilities are weighted by the fraction of character overlap with generator tokens, ensuring the total log-likelihood is preserved regardless of tokenizer granularity.
  • Training Loop: The setup is training-free for the verifier; the verifier weights are frozen. The generator is updated using group-normalized advantages derived from the CME scores.

Key Contributions

1. Novel Reward Signal: Introduction of CME as a continuous, training-free, label-free reward that relies on cross-model evaluation rather than self-referential signals.
2. Extension to Open-Ended RL: Successfully extends label-free RL to open-ended instruction following, a regime where self-consistency and majority voting fail.
3. Empirical Validation: Demonstrates that CME-GRPO improves performance across four model families (Qwen, Llama, Gemma, OLMo) and three training regimes (pretrained, SFT, instruction-tuned) without using any preference annotations.
4. Signal Verification: Proves the signal originates from cross-model evaluation rather than generic regularization by showing that a randomly initialized verifier yields significantly lower performance than real-weighted verifiers.

Experimental Results
The authors evaluated CME-GRPO on the AlpacaEval 2.0 benchmark (805 prompts) using LLM-as-Judge (GPT-5.2 and Claude Sonnet 4.6).

• Performance vs. Base: CME-GRPO outperformed the untrained base model in every head-to-head comparison across all model families. Tie-adjusted win rates ranged from 52.5% to 71.4%.
• Performance vs. SFT/DPO: On SFT backbones, CME-GRPO matched the performance of DPO models trained on the same base using the full UltraFeedback dataset (including preference annotations), despite CME-GRPO using only the prompts and no preference labels.
• Verifier Capability: Performance scaled with the capability of the verifier. A larger verifier (Gemma-4-E4B-it) yielded higher win rates than smaller ones. Crucially, a randomly initialized verifier underperformed all real verifiers, confirming the signal is not an artifact of generic regularization.
• Comparison to Self-Referential Methods: CME-GRPO consistently outperformed RENT (a self-referential entropy minimization method), particularly on pretrained backbones where self-referential signals are prone to reinforcing errors.

Significance and Claims
The paper claims that CME provides a viable path to post-training LLMs for open-ended tasks without the prohibitive cost of human preference data or the structural risks of self-referential rewards. By leveraging the "surprise" of an independent, capable model, the method aligns the generator with a high-quality distribution without requiring ground-truth correctness.

The authors note that while the method is effective, it relies on the assumption that the verifier is more competent than the generator in the target domain. They also acknowledge limitations, including the compute overhead of running a verifier alongside the generator, the potential for drift toward the verifier's style (mode collapse), and the fact that evaluations rely on LLM judges rather than human ratings. The work suggests that "unsurprising" responses to a capable model are a strong proxy for high quality in the absence of explicit labels.