💬 NLP

QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards

QUBRIC is a novel framework that overcomes the structural limitations of existing rubric-based reinforcement learning by co-designing scenario-based queries and teacher-grounded rubrics, thereby enabling effective training on open-ended tasks and achieving significant performance gains in reasoning and instruction-following benchmarks.

Original authors: Rongzhi Zhang, Rui Feng, Zhihan Zhang, Jingfeng Yang, Qingyu Yin, Xin Liu, Zixuan Zhang, Priyanka Nigam, Bing Yin, Tuo Zhao, Chao Zhang

Published 2026-06-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rongzhi Zhang, Rui Feng, Zhihan Zhang, Jingfeng Yang, Qingyu Yin, Xin Liu, Zixuan Zhang, Priyanka Nigam, Bing Yin, Tuo Zhao, Chao Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to write a great story or solve a tricky problem. In the world of Artificial Intelligence, this is called Reinforcement Learning (RL). Usually, we teach robots by giving them a clear "Yes" or "No" score. For example, in math, if the answer is "42," the robot gets a gold star. If it's "43," it gets nothing. This works great for math and coding.

But what happens when there is no single right answer? What if you ask, "How do I make a good cup of coffee?" or "Write a moral story about a lost dog"? There is no "42" here. This is where the paper QUBRIC comes in.

The Problem: The "Vague Question" Trap

The authors found a big snag in how we currently teach robots to handle these open-ended questions.

Think of it like this: You ask a student, "Explain how to bake a cake."

The Old Way: You try to make a checklist (a "rubric") for the teacher to grade the answer. But because the question is so broad, the checklist ends up being vague. "Did they mention flour?" "Did they mention eggs?" It's hard to be fair.
The Naive Fix: Someone tries to make the question more specific: "Explain how to bake a cake using the 2024 Grandmaster Baking Guide."
- The Disaster: The robot doesn't have this guide. It doesn't exist! So, the robot either refuses to answer (because it can't find the guide) or makes up a fake guide. The teacher's checklist then just checks: "Did you refuse to answer?" or "Did you lie?" The robot gets no useful feedback on how to bake a cake. It just learns to be silent or to lie.

The paper calls this the "Fabricated Reference Failure." You can't grade a robot on a rulebook that doesn't exist.

The Solution: QUBRIC (The "Co-Designer" Approach)

The authors created a new system called QUBRIC. Instead of just making a checklist for a messy question, QUBRIC changes the question and the checklist together, like a team of architects and inspectors working side-by-side.

Here is how it works, using a simple analogy:

1. The "Key Point" Detective (Rewriting the Question)

Instead of asking the robot to "Explain machine learning" (which is too huge), the system looks at what a human expert (a "Teacher") would say. It pulls out the most important, tiny facts (Key Points).

Analogy: Imagine you want to teach a child about "Animals." Instead of that, you say, "Imagine you are a zookeeper at a specific zoo with a hungry lion. The lion only eats meat from the north side of the enclosure. How do you feed it?"
Why this works: You haven't changed the topic (it's still about animals/feeding), but you've created a specific scenario with a clear answer space. You aren't asking for a fake book; you are asking for a solution to a real, contained problem.

2. The "Contrastive" Inspector (Making the Checklist)

Now, the system generates the checklist (the rubric). It doesn't just guess what a good answer looks like. It compares a "Teacher's" perfect answer against a "Student's" (the robot's) current answer.

Analogy: The inspector looks at the Teacher's answer and says, "Ah, the Teacher mentioned checking the lion's teeth before feeding. The Student forgot that."
The checklist becomes: "Did the response mention checking the lion's teeth?"
This is a Constitutive Rubric: The rule is self-contained. You don't need to know outside facts to grade it; you just check if the specific detail is there.

3. The "Difficulty Filter" (Picking the Right Practice Problems)

Not every question is good for learning.

If the question is too easy, the robot already knows the answer. No learning happens.
If it's too hard, the robot fails every time and gets confused.
QUBRIC acts like a coach who only picks practice problems where the student has a 20% to 50% chance of getting it right. This is the "sweet spot" where learning actually happens.

What Happened? (The Results)

The authors tested this system on a robot that was good at following instructions but not great at complex reasoning.

The Test: They asked the robot to handle difficult, open-ended tasks (like writing creative stories or solving complex logic puzzles) and even tasks it had never seen before (like legal reasoning or moral dilemmas).
The Result: The robot improved significantly.
- On a tough "Arena" test for instruction following, it jumped up by 5.5 points.
- On three completely different types of reasoning tests (legal, moral, narrative), it improved by an average of 6.3 points.

The Big Takeaway

The paper proves that you can't just give a robot a checklist for a vague question and expect it to learn. The question itself must be designed to be checkable.

By rewriting the question into a specific, realistic scenario (grounded in real facts, not fake books) and then building a checklist based on that specific scenario, the robot gets clear, useful feedback. It learns to think better, not just to guess or refuse to answer.

In short: QUBRIC teaches us that to teach a robot to think, you have to ask it the right kind of question first. You can't grade a student on a test that doesn't make sense.

Technical Summary: QUBRIC – Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards

1. Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has driven significant advances in domains like mathematics and coding, where ground-truth answers provide clean reward signals. However, extending RL to open-ended tasks (e.g., instruction following, legal reasoning, narrative generation) remains challenging due to the lack of verifiable answers.

Existing approaches typically treat the query distribution as fixed while optimizing rubrics (structured criteria for evaluation), or they reshape queries to create executable constraints. The authors identify a structural bottleneck: rubric quality is fundamentally constrained by query structure.

Open-ended queries yield vague rubrics that judges cannot apply consistently.
Naive narrowing (making a query specific without grounding) often introduces fabricated references (e.g., citing non-existent guidelines or documents). This causes rubrics to collapse into testing whether a model refuses to answer rather than whether it reasons correctly, resulting in uninformative reward signals where all responses fail.

The paper argues that query structure and rubric design are coupled: a query must be narrowed in a way that provides a determinate reasoning target for the rubric generator.

2. Methodology: QUBRIC

QUBRIC is a three-stage pipeline that co-designs queries and rubrics offline before training. It operates on the principle that the query must be rewritten to support constitutive rubrics (criteria that can be evaluated using only the rubric text and the response, without external knowledge).

Stage 1: Key-Point-Grounded Query Rewriting

Instead of arbitrarily narrowing open-ended queries, QUBRIC grounds the rewriting process in the actual content of high-quality answers.

Key Point Extraction: Reference responses are generated by multiple teacher models. Core "key points" (atomic knowledge components) are extracted. Queries with high cross-model disagreement (broad answer spaces) are prioritized.
Scenario Construction: The system selects 1–2 key points and rewrites the query into a scenario-based question. The scenario includes realistic context (roles, constraints, data) that makes the query self-contained. The selected key points become the determinate core answer to the new query.
Validation: A prompt-based check ensures the rewritten query does not leak the answer, maintains internal consistency, and that the key points correctly answer the new query.
Learnability Filtering: Only query–rubric pairs where the initialization policy's pass rate falls within a 20–50% corridor are retained. This heuristic ensures the training data consists of informative examples (neither too easy nor impossible).

Stage 2: Contrastive Query-Level Rubric Generation

Rubrics are generated offline by contrasting a stronger teacher response against a weaker policy response.

Contrastive Analysis: The generator identifies content present in the teacher response but absent in the policy response, as well as shared core requirements.
Constitutive Design: Rubrics are designed to be atomic, unambiguous, non-hackable, and orthogonal. They embed grading knowledge directly (e.g., "The response explains X because of Y" rather than "The response is scientifically accurate").
Weighting: Rubrics are categorized as Critical (weight 3), Important (weight 2), or Optional (weight 1) based on their importance to the task.

Stage 3: Reinforcement Learning with Rubrics

The policy is optimized using Group Relative Policy Optimization (GRPO).

Reward Formulation: The total reward $R(q, y)$ is a weighted sum of global rubrics (safety, factuality, formatting) and query-level rubrics (task-specific reasoning).
$R(q, y) = \alpha \sum_{g \in G} w_g \cdot r_g(y) + \beta \sum_{r \in R(q)} w_r \cdot r_r(q, y)$
Training: For each query, $K$ rollout responses are generated, graded independently by an LLM judge against the static rubric set, and used to compute group-relative advantages for policy updates.

3. Key Contributions

Identification of Structural Coupling: The paper demonstrates that rubric quality degrades when queries lack structure, leading to vague or fabricated-reference failures. It establishes that query structure is a prerequisite for effective rubric-based RL.
QUBRIC Framework: A novel pipeline integrating key-point-grounded query rewriting, contrastive rubric generation, and learnability filtering to produce high-quality, evaluable training pairs.
Empirical Validation: Through matched ablations, the authors show that gains are not achieved by rubric RL alone or query narrowing alone, but specifically by the co-design of both.

4. Experimental Results

The authors evaluated QUBRIC on instruction-following benchmarks, cross-domain transfer tasks, and a proprietary shopping benchmark.

Instruction Following

ArenaHard: QUBRIC achieved a +5.5 point gain over the SFT baseline (71.0 $\to$ 76.5) on hard prompts and a +7.3 point gain on creative writing.
AdvancedIF: Improved to 84.2 rubric-level accuracy.
Trade-off: A modest decrease was observed on IFEval (-1.7pp), indicating a trade-off between strict format compliance and answer quality, consistent with prior findings.
Baselines: QUBRIC outperformed "Rubric-RL with Original Queries" (68.6 vs. 76.5 on ArenaHard) and "AutoIF-RLVR" (which excels at format but degrades on open-ended quality).

Cross-Domain Transfer

Trained only on instruction-following data, QUBRIC transferred to three held-out benchmarks:

MoReBench (Moral reasoning), PLawBench (Legal reasoning), and MuSR (Narrative reasoning).
Average Gain: +6.3 points over the SFT baseline.
Significant Improvement: +8.85 points on PLawBench, suggesting the method improves structured reasoning capabilities even in unseen domains.
Comparison: Outperformed both IF-domain baselines (AutoIF-RLVR and Rubric-RL with original queries) on these transfer tasks.

Shopping Helpfulness

On a proprietary 200-query benchmark:

Query-level rubrics raised "Helpfulness-implicit" from 55.11 (Verifiable rewards only) to 67.64.
Combining global and query-level rubrics maintained high helpfulness (67.81) while mitigating side effects like duplicate recommendations and ID hallucinations introduced by query-level optimization alone.

5. Significance and Claims

The paper claims that co-designing queries and rubrics makes rubric-based RL a practical complement to RLVR for tasks without strictly verifiable answers.

Mechanism: The core insight is that naive narrowing leads to "fabricated-reference" failures where models are penalized for not refusing to answer. By grounding the narrowing in teacher-derived key points, QUBRIC creates scenarios where the answer space is narrow enough for precise, constitutive rubrics.
Generalization: The improvements are concentrated in reasoning-related dimensions, suggesting that the method helps models learn to reason rather than just memorize patterns or follow rigid formats.
Feasibility: The study validates that LLM judges can reliably grade constitutive rubrics (achieving 75–86% accuracy against reference annotations), making the approach scalable without human-in-the-loop grading for every step.

The authors conclude that while rubric-based RL has been explored, its effectiveness is limited by the quality of the underlying queries. QUBRIC demonstrates that modifying the query structure to enable precise rubric evaluation is a necessary condition for successful RL in open-ended domains.