💬 NLP

Think-with-Rubrics: From External Evaluator to Internal Reasoning Guidance

This paper introduces "Think-with-Rubrics," a novel paradigm that transforms rubrics from external evaluators into internal reasoning guidance by having LLMs generate rubrics alongside responses during training, resulting in consistent performance improvements over existing reward-based baselines.

Original authors: Jiachen Yu, Zhihao Xu, Junjie Wang, Yujiu Yang

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiachen Yu, Zhihao Xu, Junjie Wang, Yujiu Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to write a story, solve a puzzle, or follow a complex set of instructions. In the past, we taught these robots by letting them guess, then grading their work after they finished. If they got it wrong, we told them, "Here is the grade, try again." This is like a teacher handing back a test with a red "F" at the bottom but no notes on how to fix the specific mistakes.

The paper "Think-with-Rubrics" proposes a smarter way to teach. Instead of just grading the final answer, it teaches the robot to write its own grading sheet before it even starts the work.

Here is how the paper breaks this down, using simple analogies:

1. The Problem: The "Post-Game" Critic

Currently, most AI training uses Rubrics as Rewards. Think of this like a sports coach who only shows up after the game is over. The coach looks at the final score and says, "You missed the ball three times." The player learns from the score, but they didn't have the coach's checklist in their head while they were playing. They couldn't use the rules to guide their movements in real-time.

2. The Solution: The "Pre-Game" Coach

The authors introduce Think-with-Rubrics. This changes the robot's thinking process. Now, before the robot writes a single word of the answer, it must first generate a Rubric (a checklist of rules) for itself.

The Analogy: Imagine you are baking a cake.
- Old Way: You bake the cake, taste it, and then the judge says, "It's too salty and the frosting is messy." You try again next time.
- New Way (Think-with-Rubrics): Before you turn on the oven, you write a checklist: "1. Use exactly 2 cups of flour. 2. No salt. 3. Frosting must be smooth." You then bake the cake while constantly checking your own list.

By forcing the AI to write the rules first, the rules become part of its "thought process" rather than just an external grade.

3. How the Training Works (The "Double-Check" System)

The paper describes a training process with two main stages, like a student preparing for a big exam:

Stage 1: Learning the Format (SFT Warm-up)
The robot is shown examples of how to write a checklist followed by an answer. It learns the structure: <Rubric> then <Answer>. It's like teaching a student how to format their homework correctly so the teacher can read it.
Stage 2: The Reinforcement Learning (The "Coach" Phase)
This is where the magic happens. The robot generates its own checklist and answer. Then, a "Verifier" (a smart judge) checks two things:
1. The Golden Check: Did the answer match the perfect, human-made checklist? (This is the external grade).
2. The Self-Check: Did the answer actually follow the robot's own checklist? (This is the internal consistency).

The Key Insight:
The paper found that if you only use the "Golden Check" (the human grade), the robot gets better at following human rules but might still be messy in its own thinking. But if you add the "Self-Check," the robot learns to be consistent with itself.

4. The Surprising Discovery: Self-Evolution

The most exciting finding in the paper is that the robot can actually teach itself without needing a human teacher's checklist at all!

The Analogy: Imagine a student who is so good at making their own study guides and then sticking to them that they eventually get better grades than students who only rely on the teacher's answer key.
The Result: The paper shows that a model trained only to follow its own self-generated checklists (without human "Golden" checklists) performed just as well, and sometimes even better, than models trained with human checklists. This suggests the AI can "self-evolve" by getting better at making its own rules and then following them.

5. Why It Works Better

The authors explain that this method works because it fixes a specific problem: Inconsistency.
Often, an AI might think, "I need to be short," but then it writes a long paragraph. It's a disconnect between what it thinks it should do and what it actually does.

Think-with-Rubrics forces the AI to state, "I will be short," and then immediately proves it by writing a short answer.
The training rewards the AI not just for being right, but for being consistent with its own plan.

Summary of Results

The paper tested this on several benchmarks (like IFEval and IFBench) and found:

The new method consistently beat the old "Rubric-as-Reward" methods by an average of about 3.87 points.
It worked well on both large and small AI models.
It made the AI's thinking process shorter and more efficient (condensing the "thinking" into a structured checklist).

In a nutshell: The paper teaches AI to stop guessing and start planning. By making the AI write its own rulebook before it starts working, and then rewarding it for sticking to that rulebook, the AI becomes more reliable, more consistent, and capable of improving itself without constant human supervision.

Technical Summary: Think-with-Rubrics

1. Problem Statement

Current Reinforcement Learning (RL) approaches for Large Language Models (LLMs) excel in verifiable tasks where ground-truth labels exist. However, extending RL to open-ended, unverifiable instruction-following tasks remains challenging due to the lack of reliable reward signals. Existing solutions often rely on scalar rewards or monolithic generative reward models, which provide supervision that is too coarse to capture nuanced response quality.

While recent "Rubric-as-Reward" paradigms have improved this by decomposing tasks into structured, interpretable criteria, a critical limitation persists: rubrics are treated solely as external evaluators. In current frameworks, rubrics are applied post-hoc to measure performance after the model has generated a response. This design prevents rubrics from actively guiding the model's generation process or reasoning trajectory, leaving them disjointed from the policy's primary reasoning trace.

2. Methodology: Think-with-Rubrics

The authors propose THINK-WITH-RUBRICS, a novel paradigm that internalizes rubric generation as an intrinsic part of the model's reasoning process. Instead of generating a response directly, the model first constructs a self-generated rubric to define success criteria, then generates a response conditioned on those criteria.

2.1 Core Paradigm

The policy trajectory $\tau$ is factorized into two sequential steps:

Rubric Generation: The model generates a self-generated rubric $\hat{r}$ based on the input instruction $x$ : $\hat{r} \sim \pi_\theta(\hat{r} | x)$ .
Response Generation: The model generates the final answer $y$ conditioned on both the instruction and the self-generated rubric: $y \sim \pi_\theta(y | x, \hat{r})$ .

This transforms the model from a passive executor into an active deliberator that establishes rules of success before reasoning.

2.2 Training Workflow

The training process consists of two stages:

Stage 1: SFT Warm-up
To ensure the model adheres to the specific <rubric>...</rubric><answer>...</answer> format and to prevent RL from wasting steps on format compliance, the authors perform Supervised Fine-Tuning (SFT).

Teacher Distillation: Responses are distilled from Gemini-2.5-Flash on the Openrubrics dataset.
Objective: Maximize likelihood over the entire trajectory, initializing the policy with a stable structure that separates rubric construction from answer generation.

Stage 2: RL Training
The model is optimized using the DAPO algorithm with a composite reward system $R(\tau) = \alpha R_{gold} + \beta R_{self} + \gamma R_{fmt}$ .

Rubric Verifier: A specialized, lightweight verifier (distilled from Qwen3-32B-Instruct) evaluates whether a response satisfies specific criteria, outputting a weighted compliance score.
Golden Rubric Consistency Reward ( $R_{gold}$ ): Measures compliance of the final answer against the ground-truth "golden" rubric ( $r^*$ ) from Openrubrics.
Self-Generated Rubric Consistency Reward ( $R_{self}$ ): Measures compliance of the final answer against the model's own generated rubric ( $\hat{r}$ ). This enforces internal alignment.
Format Reward ( $R_{fmt}$ ): Prevents "reward hacking" (e.g., generating trivial criteria) by ensuring the rubric and answer are successfully parsed and that the number of criteria falls within a reasonable range (5–15, peaking at 10).

2.3 Key Hypothesis

The authors hypothesize that joint optimization of $R_{gold}$ and $R_{self}$ creates a feedback loop: if an answer satisfies the self-generated rubric and also scores highly on the golden rubric, the self-generated rubric is implicitly encouraged to align with the golden standard ( $S(y, \hat{r}) \uparrow \land S(y, r^*) \uparrow \Rightarrow \hat{r} \approx r^*$ ).

3. Key Contributions

Paradigm Shift: Transforms rubrics from external evaluation metrics into internal cognitive steps, allowing models to dynamically adapt criteria to specific contexts within the reasoning chain.
Unified Training Workflow: Introduces a method combining self-generated and golden rubric supervision, demonstrating that explicit supervision on rubric quality is unnecessary; rubric learning emerges through joint answer consistency optimization.
Scalable Self-Evolution: Demonstrates that models can achieve competitive performance using only self-generated rubric supervision, suggesting a path toward scalable self-evolution without reliance on external annotations.

4. Experimental Results

Experiments were conducted on Qwen3-8B and Qwen3-4B models across three instruction-following benchmarks: IFEval, IFBench, and InfoBench.

Performance Gains: THINK-WITH-Rubrics consistently outperforms the "Rubric-as-Reward" baseline (which uses golden rubrics only as external rewards).
- On the 8B model, the mixed reward setting achieved an average improvement of 3.87 points over the baseline.
- On the 4B model, the improvement was 4.16 points.
Reward Composition: The mixed reward setting (combining golden and self-generated consistency) yielded the best results. However, the "Self-generated reward only" setting (57.17% average) slightly outperformed the "Golden reward only" setting (56.81%), validating the feasibility of self-evolution.
Efficiency: The average thinking length for THINK-WITH-RUBRICS (231 tokens) was significantly shorter than vanilla thinking (401 tokens), indicating that rubrics condense vital requirements into a streamlined process.

Mechanism Analysis

Implicit Quality Improvement: Supervision via golden rubrics implicitly improves the quality of self-generated rubrics, even without direct rubric-level comparison.
Internal Consistency: Self-generated rubric supervision significantly improves the consistency between the model's own rubrics and its answers. Untrained models often fail to follow their own generated criteria; self-supervision mitigates this internal inconsistency.
Rubric Quality vs. Performance: While self-only supervision improves instruction-following performance, the resulting rubrics are not necessarily better general-purpose evaluators (as seen in RewardBench-2). Their value lies in maintaining constraint consistency during the model's own generation process.

5. Significance and Claims

The paper claims that THINK-WITH-RUBRICS offers a robust solution for aligning LLMs with open-ended instructions by bridging the gap between reward modeling and policy generation.

Active Guidance: By treating rubrics as internal reasoning steps rather than external checks, the model actively guides its own generation toward desired outcomes, mimicking human metacognition.
Self-Evolution Potential: The ability of the model to improve using only self-generated rubrics suggests a viable path for scalable self-evolution, reducing dependence on external ground-truth annotations.
Generalizability: The paradigm remains effective across different model scales (4B to 8B) and various reward weight configurations, demonstrating robustness.

The authors conclude that integrating rubric generation into the reasoning trajectory is a general paradigm for open-ended task alignment, offering a more structured and self-consistent approach than traditional "think" or "chain-of-thought" methods.