Technical Summary: EVOLM – Self-Evolving Language Models through Co-Evolved Discriminative Rubrics

1. Problem Statement

Current post-training methods for Large Language Models (LLMs), particularly Reinforcement Learning from Human Feedback (RLHF), rely heavily on external supervision to generate reward signals. These sources include human preference annotations, proprietary API-based judges (LLM-as-a-judge), scalar reward models, or verifiable ground-truth answers. Each imposes a fundamental ceiling on model improvement: human judgment cannot supervise capabilities beyond its own; proprietary APIs create dependencies and cost barriers; scalar reward models are susceptible to reward hacking; and verifiable rewards are restricted to domains with ground-truth answers.

While LLMs encode substantial evaluative knowledge from pretraining, current methods fail to structure this latent knowledge into a reliable, self-sustaining training signal. The central challenge is to extract and structure this evaluative capacity into a form that enables reliable scoring without external supervision, allowing the model to self-improve beyond the limits of static evaluation criteria.

2. Methodology: EVOLM

The authors propose EVOLM, a post-training framework that enables a single language model to co-evolve its own generation (policy) and evaluation (rubric) capabilities. The core innovation is the explicit structuring of evaluative knowledge into discriminative rubrics—natural language criteria optimized to help a small, frozen judge distinguish between preferred and dispreferred responses.

2.1 Core Components

• Policy Model ($\pi_\theta$): Generates responses to questions.
• Rubric Generator ($\rho_\phi$): Produces instance-specific evaluation criteria (rubrics) in natural language for a given question.
• Frozen Judge ($J$): A small, fixed language model (e.g., Qwen3-1.7B) that scores responses against the generated rubrics. The judge's parameters remain frozen throughout training.

2.2 Training Objective and Mechanism

EVOLM treats the rubric as a latent variable and formalizes rubric generation via variational inference. The goal is to maximize the discriminative utility of the rubric: a rubric is "good" if, when applied by the frozen judge, it assigns a higher score to the preferred response ($a^+$) than to the dispreferred response ($a^-$).

The training objective maximizes the Evidence Lower Bound (ELBO), rewarding rubrics that widen the score gap between preferred and dispreferred responses:
$$\mathcal{L}_{rubric}(\phi) = \mathbb{E}_{r \sim \rho_\phi(r|q)} [\log \sigma(J(q, r, a^+) - J(q, r, a^-))] - \text{KL}(\rho_\phi(r|q) \| \rho_{ref}(r|q))$$
In practice, this is optimized using a margin-based reward combined with a format reward to ensure the rubric adheres to a structured JSON schema parseable by the judge.

2.3 Co-Evolving Training Loop

The method alternates between two phases using the GRPO (Group Relative Policy Optimization) algorithm:

1. Phase 1 (Policy Update): The rubric generator is frozen. The policy generates responses, which are scored by the frozen judge using the current rubrics. The policy is updated to maximize these rubric-derived rewards.
2. Phase 2 (Rubric Update): The policy is frozen. Preference pairs $(a^+, a^-)$ are constructed from the policy's own outputs. The rubric generator produces candidate rubrics, which are scored by the judge on both responses. The rubric generator is updated to maximize the score margin between the preferred and dispreferred responses.

2.4 Preference Pair Construction

To avoid external labels, EVOLM constructs preference pairs entirely from the policy's own outputs using three complementary methods:

• Temporal Contrast: Pairs a current response ($a^+$) with an earlier response ($a^-$) from a previous training checkpoint. As the policy improves, the temporal gap creates a natural curriculum of increasing difficulty.
• Inferred Question (IQ): The policy infers the question a preferred response addresses, then generates a dispreferred response to that inferred question to test relevance.
• Rubric-Conditioned (RC): Generates a preferred response conditioned on the rubric and a dispreferred response conditioned only on the question, testing the rubric's utility.

3. Key Contributions

• Self-Evolving Framework: EVOLM is the first method to formally define rubric quality as discriminative utility and train a rubric generator end-to-end without external supervision, proprietary APIs, or task-specific verifiers.
• Co-Evolution: The framework establishes a feedback loop where the policy's improvement forces the rubrics to sharpen (to remain discriminative), and sharper rubrics provide more informative rewards to drive further policy improvement.
• Rubric Enrichment Mechanism: Qualitative analysis reveals that trained rubrics evolve from abstract labels to concrete, verifiable checks. They embed specific expected values (e.g., "the answer is 144") and decompose constraints into individually checkable sub-tasks, transforming holistic judgment into pattern matching that even small judges can perform reliably.
• Generalization: The learned rubrics transfer effectively to unseen policies, different model families, and out-of-distribution domains (e.g., deep research tasks) without retraining.

4. Experimental Results

Experiments were conducted using Qwen3-8B as the policy/rubric generator and Qwen3-1.7B as the frozen judge.

• Downstream Performance: The EVOLM-trained policy achieved an average of 69.3% on the OLMo3-Adapt suite (12 benchmarks), outperforming:
  • Policies trained with GPT-4.1 prompted rubrics (66.7%).
  • Policies trained with Qwen3-8B prompted rubrics (67.5%).
  • State-of-the-art scalar reward models (SkyWork-RM-V2) at 59.7%.
  • Other rubric-based RL methods (RAR, RRD, RLCER, RUBRIC-ARM) which ranged from 66.7% to 67.6%.
• Rubric Quality: The learned rubrics outperformed GPT-4.1 on RewardBench-2 by 25.7% (46.0% vs. 20.3% in specific metrics, though the abstract cites a 25.7% improvement over GPT-4.1 rubrics on the benchmark).
• Generalization:
  • Cross-Policy: Rubrics trained on Qwen3-8B improved downstream performance on unseen Qwen3-4B and Llama-3.1-8B policies compared to GPT-4.1 prompted rubrics.
  • Cross-Judge: Rubrics trained with a 1.7B judge remained effective when paired with larger judges (Qwen3-8B, OLMo-3-7B), showing significant gains over prompted baselines.
  • Out-of-Distribution: On deep research tasks (HealthBench, ResearchQA), EVOLM rubrics achieved higher pairwise agreement with expert human rubrics than GPT-4.1 (58.4% vs. 52.5% on HealthBench).
• Ablation Studies:
  • Judge Size: A small 1.7B judge produced the best downstream policy, suggesting that rubric content quality is more critical than judge capability.
  • Training Dynamics: Co-evolving training outperformed sequential training (training rubrics then policy), highlighting the importance of the adaptive feedback loop.
  • Reward Overoptimization: The scalar reward model, despite dominating static benchmarks (RewardBench-2: 86.4%), produced the weakest downstream policy, confirming that static criteria fail to adapt to the shifting policy distribution.

5. Significance and Claims

The paper claims that EVOLM demonstrates that structuring a model's evaluative capacity into co-evolving discriminative rubrics enables self-improvement without external supervision.

The authors argue that every external reward source imposes a ceiling: human judgment cannot supervise beyond its own capacity, and static criteria cannot adapt to the output distribution a learning policy creates. By co-evolving the evaluation criteria with the policy, EVOLM removes these constraints. The learned evaluation structure is shown to be transferable across unseen policies, judges, and domains, suggesting a path toward autonomous, scalable self-improvement in language models.

The work positions rubrics not just as a tool for interpretability, but as a mechanism to extract latent evaluative knowledge from pretraining and structure it into a reliable, self-sustaining training signal.