Technical Summary: G-Zero

1. Problem Statement

Self-evolving Large Language Models (LLMs) have demonstrated significant success in verifiable domains (e.g., mathematics, code) where programmatic oracles provide ground truth for Reinforcement Learning from Verifiable Rewards (RLVR). However, these models struggle in open-ended domains (e.g., creative writing, dialogue, instruction following) where objective verifiers do not exist.

Current approaches for these unverifiable tasks rely heavily on LLM-as-a-Judge mechanisms to generate surrogate reward signals. The paper identifies two critical limitations in this paradigm:

1. Capability Bottlenecks: The performance ceiling of the evolving model is fundamentally limited by the capabilities of the judge model.
2. Reward Hacking: Models tend to exploit the judge's stylistic vulnerabilities (e.g., bias, formatting preferences, verbosity) rather than genuinely improving response quality.

The central question addressed is: How can self-evolution scale in unverifiable domains without relying on external judges or internalizing the pathologies of proxy reward signals?

2. Methodology: The G-Zero Framework

G-Zero is a verifier-free, co-evolutionary framework that derives supervision entirely from internal distributional dynamics. It operates through the interaction of two models: a Proposer ($\pi_P$) and a Generator ($\pi_G$).

2.1 The Intrinsic Signal: Hint-$\delta$

The core innovation is Hint-$\delta$, an intrinsic reward signal that quantifies the predictive shift in the Generator's distribution when conditioned on a self-generated hint.

• Mechanism: For a query $q$, the Generator produces an unassisted response $a_{hard}$. A hint $h$ is then provided, and the Generator produces a hint-conditioned response $a_{assisted}$.
• Calculation: Hint-$\delta$ is the per-token mean log-likelihood difference between the unassisted and hint-conditioned distributions:
  $$\delta(q, h, a_{hard}) = \frac{1}{T} \sum_{t=1}^{T} \left[ \log \pi_G(a_t | q, a_{<t}) - \log \pi_G(a_t | q, h, a_{<t}) \right]$$
• Properties: A large $\delta$ value occurs only when the query is genuinely challenging for the Generator (unassisted response is flawed) AND the hint provides necessary, missing information. This signal is invariant to response length and computed entirely from the Generator's own log-probabilities, eliminating external judge bias.

2.2 Co-Evolutionary Loop

The framework operates in iterative rounds involving two phases:

Phase 1: Proposer Training (GRPO)

• Goal: Train $\pi_P$ to synthesize challenging queries ($q$) paired with informative hints ($h$) that maximize the Generator's $\delta$.
• Optimization: Uses Group Relative Policy Optimization (GRPO).
• Reward Function: $r(q, h) = \delta(q, h, a_{hard}) - P_{length} - P_{BLEU}$.
  • $P_{length}$: Penalizes hints exceeding a character budget (~200 chars) to prevent verbose reward hacking.
  • $P_{BLEU}$: Penalizes repetitive query generation to ensure exploration coverage.
• Outcome: The Proposer targets the Generator's "blind spots," creating a curriculum of increasingly difficult tasks.

Phase 2: Generator Training (DPO)

• Goal: Train $\pi_G$ to internalize the structural and logical improvements suggested by the hints.
• Data Curation:
  • The Proposer generates $(q, h)$ pairs.
  • The Generator produces $a_{hard}$ (no hint) and $a_{assisted}$ (with hint).
  • Filtering Strategy: Unlike the Proposer which seeks high $\delta$, the Generator training dataset ($D_{R+1}$) retains pairs in the lower half of the $\delta$ distribution.
    • Rationale: High-$\delta$ pairs often represent "answer leakage" (hints giving away the solution), violating DPO's implicit KL constraints. Lower-$\delta$ pairs represent "hard-to-distinguish" signals where the hint induces subtle, structural refinements rather than trivial corrections.
• Optimization: Uses Direct Preference Optimization (DPO) with a length-normalized loss.
  • Chosen ($y_w$): $a_{assisted}$ (hint-guided).
  • Rejected ($y_l$): $a_{hard}$ (unassisted).
  • Reference Model: A frozen snapshot of $\pi_G$ from the start of the round.

3. Theoretical Analysis

The paper provides a theoretical guarantee for an idealized standard-DPO variant of G-Zero.

• Assumptions: The framework assumes a linear reward structure, a Bradley-Terry preference model, and that the Proposer induces sufficient exploration coverage (via the BLEU reward) while the $\delta$-filter keeps pseudo-label noise ($\eta_\delta$) low.
• Result (Theorem 1): Under these conditions, there exists an iterate $t_0$ such that the suboptimality gap $J(\pi^*) - J(\pi_{t_0})$ is bounded by terms related to the sample complexity and the noise level. Specifically, the bound depends on the Proposer-induced coverage and the $\delta$-certified pseudo-label score noise. This proves that the co-evolutionary loop can converge to a near-optimal policy without external verifiers.

4. Experimental Results

Experiments were conducted on Qwen3-8B-Base and Llama-3.1-8B-Instruct across two rounds of self-evolution.

• Open-Ended Tasks:
  • AlpacaEval 2.0 (Chat): G-Zero achieved substantial gains. For Llama-3.1, the length-controlled win rate increased from 24.12% (base) to 27.86% (Round 2).
  • IFEval (Instruction Following): Improvements were observed in strict and loose instruction adherence across both models.
• Verifiable Tasks (Transfer):
  • Math (AIME24/AIME25): Despite being trained primarily on open-ended, non-verifiable data, G-Zero showed significant reasoning improvements.
    • Qwen3-8B-Base: AIME25 score jumped from 7.19% to 12.40% after Round 2.
    • Llama-3.1-8B-Instruct: AIME25 improved from 0.42% to 0.63% (noting the base was very low, but gains were consistent).
• Comparison with R-Zero:
  • R-Zero (a verifier-based method) showed strong gains in math but suffered from capability collapse in conversational and instruction-following tasks (e.g., Qwen3 Chat dropped from 8.94% to 8.04%).
  • G-Zero maintained or improved performance across all metrics (Chat, IFEval, Math), demonstrating robust generalizability without the trade-offs seen in judge-based methods.

Key Analysis Findings:

• Structural Transfer: The training data was dominated (>70%) by non-verifiable tasks (advice, writing). The reasoning improvements in math were attributed to the internalization of logical depth and compositional complexity from these open-ended tasks, rather than domain-specific memorization.
• Co-Evolutionary Arms Race: The distribution of Hint-$\delta$ shifted rightward in Round 2, indicating that as the Generator improved, the Proposer successfully synthesized more complex and impactful hints to maintain the learning signal.

5. Significance and Contributions

The paper claims three primary contributions:

1. A Verifier-Free Co-Evolutionary Framework: G-Zero enables continuous self-improvement in open-ended domains by replacing external judges with an intrinsic signal (Hint-$\delta$) derived from the model's own predictive shifts.
2. Theoretical Characterization: The authors formalize the co-evolutionary loop and prove a best-iterate suboptimality guarantee, showing that self-evolution is theoretically sound provided the Proposer ensures exploration and the filter maintains data quality.
3. Empirical Validation of Cross-Domain Transfer: The work demonstrates that models can internalize logical reasoning patterns from non-verifiable, open-ended tasks and successfully transfer them to rigorous, verifiable domains like mathematics. This challenges the notion that self-evolution requires verifiable oracles or external judges to scale.

Conclusion:
G-Zero provides a scalable, robust pathway for autonomous LLM self-evolution. By deriving supervision from internal distributional dynamics, it bypasses the capability ceilings and reward hacking vulnerabilities of external judges, proving that models can elevate their capabilities using only intrinsic feedback.