Technical Summary: Rubric-Guided Self-Distillation (RGSD)

Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has driven significant post-training progress for tasks with programmatic verifiers (e.g., math, code). However, for open-ended domains such as clinical advice, scientific exposition, and creative writing, programmatic verification is inapplicable. The community has adopted rubrics—weighted checklists of criteria—as a proxy for ground truth.

Current rubric-based training methods, such as Group Relative Policy Optimization (GRPO), rely on an external Large Language Model (LLM) verifier to score every generated rollout against the rubric. This approach introduces three critical limitations:

1. Computational Overhead: Every training step requires expensive LLM judge calls (often $G \times K$ calls per prompt, where $G$ is the number of rollouts and $K$ is the number of criteria), creating latency and rate-limit bottlenecks.
2. Verifier Bias and Reward Hacking: The reward signal inherits the idiosyncrasies and biases of the specific verifier model. Policies may learn to "hack" the verifier (e.g., by increasing verbosity or making unsupported claims) rather than genuinely satisfying the criteria.
3. Sparse Credit Assignment: Rubric-based RL typically reduces a multi-criterion assessment to a single scalar reward at the end of a trajectory. This provides a weak learning signal for determining which specific token-level decisions improved or harmed rubric satisfaction.

Methodology: Rubric-Guided Self-Distillation (RGSD)

The authors propose Rubric-Guided Self-Distillation (RGSD), a verifier-free training method that eliminates the external LLM judge from the training loop. The core insight is empirical: when a base model is provided with the rubric as privileged context, its responses score significantly higher (30–45 percentage points) than when it is unconditioned. RGSD leverages this "conditioning lift" to create a dense, token-level supervision signal.

The RGSD Pipeline

RGSD instantiates two copies of the same base model ($\theta_{base}$):

1. Student ($\pi_S$): A trainable policy conditioned only on the user prompt $q$. It generates an on-policy rollout $y \sim \pi_S(\cdot|q)$.
2. Teacher ($\pi_T$): A frozen copy of the base model conditioned on the prompt $q$ and the rubric set $R$. It observes the student's generated prefix $y_{<t}$ and the rubric to produce a target token distribution $\pi_T(\cdot|q, R, y_{<t})$.

Training Objective:
Instead of optimizing for a scalar reward, the student is trained to match the teacher's per-token distribution using a clipped Jensen–Shannon Divergence (JSD):
$$L_{RGSD}(\theta_S) = \mathbb{E}_{(q,R) \sim D, y \sim \pi_S} \left[ \frac{1}{T} \sum_{t=1}^{T} D_{\beta}^{clip} \left( \pi_S(\cdot|q, y_{<t}) \parallel \pi_T(\cdot|q, R, y_{<t}) \right) \right]$$
where $\beta=0.5$ interpolates between forward and reverse KL.

Key Implementation Details:

• Dense Supervision: Unlike GRPO's sparse trajectory-level reward, RGSD provides a learning signal for every token in the rollout.
• No Verifier Calls: The training loop requires zero external judge calls; supervision comes entirely from the frozen teacher.
• Reasoning Trace Handling: For reasoning models (e.g., Qwen3-Thinking), the rubric-conditioned teacher may explicitly reference rubric criteria in its internal "thinking" trace ($\dots$). To prevent the student from learning to fabricate these references (which it cannot see at inference), RGSD masks the thinking-block tokens during loss calculation, distilling only the final response tokens.

Key Contributions

1. A Verifier-Free Training Algorithm: RGSD is the first method to use rubrics as a privileged context for on-policy self-distillation, removing the LLM judge from the optimization loop entirely.
2. Competitive Performance with Lower Cost: Across four base models (Qwen-2.5 3B/7B, Qwen3-Thinking 4B/8B) and two domains (Medical, Science), RGSD achieves rubric satisfaction scores comparable to judge-based GRPO.
   • Medical Domain: RGSD improves base scores by +6.1pp vs. GRPO's +5.9pp.
   • Science Domain: RGSD improves by +4.9pp vs. GRPO's +4.5pp.
3. Efficiency and Conciseness: RGSD requires only one on-policy rollout per prompt (vs. 16 for GRPO) and zero judge calls. Furthermore, on Qwen-2.5 models, RGSD achieves comparable quality with substantially shorter responses, avoiding the "verbosity drift" observed in GRPO.
4. Ablation Insights:
   • Enrichment Signal: Using the raw rubric as the teacher's context provides a stronger enrichment signal than using a self-generated "golden" reference response.
   • Judge Strength: While a stronger verifier (e.g., gpt-oss-120b) can allow GRPO to outperform RGSD in specific settings, this comes at the cost of retaining the expensive per-rollout verifier loop and often producing longer, more claim-dense responses.

Results and Analysis

• Performance Parity: RGSD is competitive with GRPO across in-distribution (RubricHub) and out-of-distribution (HealthBench, ResearchQA) benchmarks. The authors interpret sub-1pp differences as parity rather than significant wins, noting that seed variance is low.
• Response Length and Factuality:
  • On Qwen-2.5 models, GRPO exhibits severe verbosity drift, generating responses 1.4–2.3× longer than RGSD without consistent score gains.
  • A factual-claim audit on HealthBench reveals that GRPO's length inflation correlates with a higher rate of unsupported claims (false-claim rate rising from 30.5% to 45.1%), whereas RGSD maintains a more conservative claim density (rising to 35.1%).
• Conditioning Gap as a Diagnostic: The magnitude of the "rubric-conditioning gap" (the performance lift when a base model sees the rubric) serves as a predictor for RGSD's effectiveness. In domains where the base model benefits significantly from seeing the rubric (e.g., Medical), RGSD recovers a larger fraction of that potential.

Significance and Positioning

The paper positions RGSD not as a universal replacement for verifier-based RL, but as a complementary regime.

• When to use RGSD: It is attractive when verifier calls are computationally expensive, operationally difficult, or unreliable, and when the base model exhibits a strong rubric-conditioning response. It effectively internalizes rubric knowledge without the risk of reward hacking associated with external judges.
• When to use GRPO: If the primary goal is maximizing rubric scores and a strong, reliable verifier is available and affordable, judge-based GRPO (potentially with a stronger judge) may still achieve higher performance, albeit at a higher cost and with longer, potentially less factual responses.

The authors conclude that RGSD demonstrates that rubrics need not be used solely as external grading instruments; they can serve as privileged context for dense self-distillation, offering a scalable path to post-training in open-ended domains.