Technical Summary: Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation

Problem Statement

Post-training of reasoning language models (LLMs) currently relies heavily on two paradigms, both of which suffer from significant limitations in providing fine-grained feedback:

1. Reinforcement Learning with Verifiable Rewards (RLVR): Methods like Group Relative Policy Optimization (GRPO) typically compress evaluative feedback into a sparse, scalar signal (e.g., binary correctness or a single score). While effective for verifiable tasks, this approach obscures why a trajectory succeeded or failed, creating a "credit-assignment bottleneck" where the model cannot distinguish between correct and incorrect intermediate reasoning steps.
2. On-Policy Self-Distillation (OPSD): These methods replace sparse rewards with dense, token-level guidance from a teacher model. However, existing approaches condition the teacher on a single privileged reference trajectory (a specific gold solution). This ties supervision to a specific reasoning path, potentially over-specifying how an answer must be produced and ignoring the space of valid alternative derivations. Furthermore, if the student deviates slightly from this single path, the teacher may encourage global revision rather than localized correction.

Additionally, while rubrics (structured evaluation criteria) offer a richer, more interpretable representation of quality than scalar scores, prior work has largely discarded this structure during optimization by aggregating rubric judgments into a single scalar reward.

Methodology: Rubric-Conditioned Self-Distillation (RCSD)

The authors propose Rubric-Conditioned Self-Distillation (RCSD), a framework that treats rubrics as privileged, structured teacher-side supervision for on-policy self-distillation. Instead of collapsing rubric feedback into a scalar, RCSD conditions the teacher model on criterion-level rubric information to provide dense, token-level guidance on the student's own sampled trajectories.

The framework operates via a two-stage pipeline:

Stage I: Learning a Rubric Generator

To address the high cost of obtaining instance-specific rubrics, the authors train a dedicated rubric generator.

• Teacher Policy: Receives the question ($x$) and a reference answer ($y^\star$) as privileged input to infer the appropriate evaluation criteria.
• Student Policy: Observes only the question ($x$) and learns to generate a rubric ($r$) that approximates the teacher's criteria.
• Objective: The student is trained via on-policy distillation to match the teacher's token distribution along the student's own sampled rubric trajectory. This amortizes the generation of instance-specific criteria into a reusable model.

Stage II: Rubric-Conditioned Reasoning

The trained rubric generator is used to guide the training of the reasoning model.

• Teacher Policy: Receives the question ($x$) and the generated rubric ($r$) as privileged context.
• Student Policy: Observes only the question ($x$) and generates a reasoning trajectory ($\hat{y}$).
• Objective: The student minimizes the Kullback-Leibler (KL) divergence between its distribution and the teacher's distribution, where the teacher is conditioned on the rubric.
  $$\mathcal{L}_{reason} = \mathbb{E}_{\hat{y} \sim p_S} \left[ \sum_{t=1}^{|\hat{y}|} D_{KL} \left( p_T(\cdot \mid x, r, \hat{y}_{<t}) \parallel p_S(\cdot \mid x, \hat{y}_{<t}) \right) \right]$$
• Key Design: By conditioning on the rubric rather than a single reference answer, the teacher provides criterion-aware guidance. This allows the optimization signal to distinguish different dimensions of partial correctness (e.g., a correct step followed by a local error) rather than compressing them into a single score or a single reference path. The authors utilize Forward KL divergence to encourage the student to cover the support of the rubric-conditioned teacher distribution, preserving diversity across valid reasoning paths.

Key Contributions

1. New Supervision Interface: RCSD reframes rubrics not merely as scoring tools but as structured supervision interfaces that shape token-level learning during optimization.
2. Two-Stage Pipeline: The authors introduce a scalable method to learn task-specific rubrics automatically (Stage I) and then use them to guide reasoning (Stage II), removing the dependency on manual rubric annotation or expensive LLM-as-a-Judge rewards during the distillation phase.
3. Criterion-Aware On-Policy Learning: Unlike standard OPSD which conditions on a single trajectory, RCSD conditions on criteria, enabling fine-grained credit assignment that penalizes specific local errors while preserving correct steps, even on the student's own sampled trajectories.

Experimental Results

The method was evaluated on a diverse suite of science reasoning benchmarks, including verifiable tasks (GPQA-DIAMOND, SciBench, PIQA) and non-verifiable, open-ended tasks (ResearchQA, RubricHub).

• Performance: RCSD achieved the best overall average score (70.6) across benchmarks. It outperformed the base Qwen3-8B model by 4.7 points, surpassing GRPO by 1.0 point and OPSD by 0.9 points on average.
• Domain Specificity: Gains were most pronounced on rubric-based and scientific reasoning tasks (e.g., +8.2 points on ResearchQA, +4.9 points on RubricHub), where scalar rewards often fail to capture response quality.
• Generalization: The method demonstrated robust generalization to out-of-domain medical benchmarks (MedMCQA, PubMedQA) without catastrophic forgetting, suggesting the learned criteria capture general reasoning structures.
• Ablation Studies:
  • Loss Type: Forward KL was found to be the most effective objective, yielding the highest average performance and maintaining higher student entropy (diversity) compared to Reverse KL or JS Divergence.
  • Rubric Source: Using rubrics generated by the learned Stage-I generator yielded results competitive with using ground-truth reference rubrics, validating the scalability of the two-stage approach.
  • Robustness: The method remained effective even with degraded rubric inputs (generic, noisy, or reduced), though instance-specific learned rubrics provided the strongest supervision.

Significance and Claims

The paper claims that RCSD offers a verifier-free approach to post-training open-ended reasoning models. By preserving criterion-level structure during optimization and converting rubrics into dense token-level guidance, the method addresses the credit-assignment bottleneck inherent in scalar reward optimization and the path-specific bias of reference-conditioned distillation.

The authors posit that RCSD is particularly significant for tasks where high-quality responses cannot be fully captured by automatic verification (exact-match answers or unit tests) or scalar outcome rewards. By enabling the model to learn what properties a response should satisfy (via rubrics) rather than how to mimic a specific solution path, RCSD facilitates more flexible and robust reasoning improvements. The work suggests that structured, multi-dimensional feedback is a superior supervision signal for complex reasoning tasks compared to both sparse rewards and single-trajectory distillation.