Rubrics as Privileged Information for Open-Ended Generation

Imagine you are trying to teach a robot how to write a perfect story or give helpful medical advice. In the world of artificial intelligence, this is called "open-ended generation." Unlike math problems where there is only one right answer (like 2+2=4), these tasks have millions of ways to be "good." A story can be funny, sad, or thrilling, and still be excellent. The big challenge for scientists is: how do you teach a robot to find those millions of good paths without getting lost?

Traditionally, researchers have used two main tricks. The first is "Reinforcement Learning," which is like a video game where the robot gets a single point at the very end of the level for doing well. The second is "Distillation," where a smart teacher shows a student exactly what to write, and the student tries to copy it. But here's the catch: in open-ended tasks, copying just one example might be too strict. If the teacher writes a story about a dragon, the student might think they must write about a dragon, missing the fact that a story about a wizard would have been just as good. This paper explores a new way to teach robots by giving them a "rubric"—a checklist of what makes a good answer—instead of just a single example or a single score at the end.

The Paper's Big Idea: The Checklist vs. The Single Answer

The authors, working at Apple, propose a method they call RuPI (Rubrics as Privileged Information). They wanted to see if they could teach AI models to be better at open-ended tasks by using detailed checklists (rubrics) as a special "guide" for the teacher, but not for the student.

Think of it like a cooking class.

• The Old Way (Reference Completion): The teacher shows the student a photo of a perfect lasagna and says, "Copy this exactly." The student tries to mimic the photo. But what if the student's kitchen has different ingredients? They might get stuck trying to force a photo into reality, or they might miss the fact that a different kind of pasta could also be delicious.
• The Reinforcement Learning Way (Rubric as Reward): The student cooks a meal, and a judge tastes it and gives a single number, like "8 out of 10." The student tries again, hoping to get a 9. But the judge doesn't tell them what was wrong, just that it wasn't perfect.
• The New Way (RuPI): The teacher has a secret checklist (the rubric) that says, "The dish must have garlic, it shouldn't be too salty, and it needs a creamy sauce." The teacher uses this checklist to guide their own cooking, creating a "perfect" version in their head. The student, however, only sees the ingredients and the final dish the teacher made, without seeing the checklist. The student learns to match the teacher's high-quality cooking style, which was guided by the principles of the checklist, not just a single photo.

What They Found

The researchers tested this idea on two very different types of questions: medical advice (using a dataset called HealthBench) and science questions (using RubricHub). They used three different AI models (Qwen and Llama families) to see if the method worked.

Here is the surprising result: The checklist (rubric) was a much better teacher than the single perfect answer (reference).

When the teacher was guided by the checklist, the student learned much faster and became better at the task. In fact, the checklist method beat the "single photo" method by a significant margin. On the medical tests, the checklist method improved scores by up to 0.10 points compared to the reinforcement learning method, and by 0.034 to 0.079 points compared to the single-answer method.

Why did this happen? The authors explain that a single "perfect" answer is just one tiny point in a huge ocean of possible good answers. If you force a student to copy just that one point, you limit their creativity and ability to handle new situations. But a checklist describes the entire ocean of good answers. It tells the student, "Stay within these boundaries," rather than "Stand exactly here." This gives the student a much stronger and more flexible signal to learn from.

The "On-Policy" Secret Sauce

There was another crucial discovery. The method only worked when the student was learning from its own attempts (called "on-policy" roll-outs). If the student tried to learn from the teacher's pre-made answers (off-policy), the magic disappeared.

Imagine a dance instructor. If the student practices their own moves and the instructor corrects them based on a checklist, the student improves. But if the student just watches a video of the instructor dancing perfectly without ever trying to move their own feet, they don't learn as well. The paper found that the checklist only becomes a powerful teacher when it is used to correct the student's own mistakes in real-time.

Does It Break Anything Else?

The researchers were worried that teaching the robot to be great at medical advice might make it forget how to do math or follow simple instructions. They tested the models on general knowledge tests (like MMLU) and math problems (GSM8K). The results were reassuring: the models got better at the specific task without losing their general smarts. However, they did find that using a specific type of mathematical correction (called "forward KL") sometimes made the models slightly worse at following instructions, so they recommend using a different correction method ("reverse KL") to keep the models safe and helpful.

The Bottom Line

This paper suggests that for tasks where there isn't just one right answer, we should stop trying to force AI to copy a single "perfect" example or chase a single score. Instead, we should give the AI a clear set of rules (a rubric) to guide its learning. By using these rules as a "privileged" guide for the teacher, the student learns to navigate the vast space of good answers much more effectively. It's a shift from teaching by example to teaching by principle, and the results show it makes the AI smarter, more flexible, and better at handling the messy, open-ended problems of the real world.