🤖 AI

Inducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent Evaluation

The paper introduces RubricForge, a method that evolves human-readable judging rubrics from ground-truth trajectories to significantly reduce over-crediting of failed agent behaviors in reward-free evaluations, prioritizing faithfulness and false-pass reduction over raw agreement with generic baselines.

Original authors: Darragh Quinn, David Dylan, Roisin Healy, Fionn Carroll, Maeve Donnelly, Cormac Sheehan

Published 2026-08-17

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Darragh Quinn, David Dylan, Roisin Healy, Fionn Carroll, Maeve Donnelly, Cormac Sheehan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where we build digital assistants—smart agents that can book flights, shop for clothes, or write code by talking to computers and websites. These agents are getting incredibly good at sounding confident and polite. But here's the tricky part: how do we know if they actually did the job, or if they just gave a really smooth, convincing story about doing it? In the past, we had to check every single job manually, like a teacher grading every essay by hand. But now, there are too many agents to check one by one. So, scientists started using a second AI to act as a "judge" to grade the first one. This is like hiring a robot teacher to grade the work of another robot. The problem is, these robot teachers are often easily fooled. They love a good story. If an agent writes a long, fluent, confident-sounding report but actually failed to book the flight, the judge might still give it an "A" just because the writing was so nice. This is dangerous because it means we might ship broken software that looks perfect on paper but crashes in real life.

This paper introduces a clever new way to train these robot judges so they stop falling for the "smooth talker" trap. Instead of just telling the judge what to look for with a generic rulebook, or tweaking the judge's brain (which is expensive and hard to understand), the authors created a method called RUBRICFORGE. Think of it like a detective training a new partner. Instead of just giving the partner a list of rules, they show them a small pile of real cases where they know exactly who got away with a crime and who didn't. The detective then asks the partner to write a new set of rules based on those specific cases, and they keep refining those rules until the partner can spot the fakes perfectly. The result is a set of instructions written in plain English that the judge follows. The big surprise? This new method doesn't necessarily make the judge agree with the "truth" more often on simple yes/no questions. Instead, it makes the judge much better at catching the specific kind of mistake that matters most: giving a passing grade to a failure that just sounded really good.

The Story of the "Smooth-Talking" Robot

In the world of artificial intelligence, we have these "agents" that act like digital employees. They try to do tasks like buying a shirt or fixing a bug. To see if they did a good job, we usually have a "ground truth"—a perfect, unchangeable answer key. For example, if the agent was supposed to buy a red shirt, the ground truth is a database check: "Did a red shirt get added to the cart?"

But checking that database is slow, expensive, or sometimes impossible if the task involves real money or dangerous actions. So, we use a second AI, a "Judge," to look at the agent's conversation and guess if it succeeded. The problem is that these Judges are biased. They are like a teacher who gives an A to a student who writes a beautiful essay but gets the math wrong. They prefer answers that are fluent, confident, and long, even if the actual task failed. This is called "over-crediting." It's a silent killer because if you think your agent is working because the Judge said "Pass," but it's actually broken, you might release it to the public and cause a disaster.

The Solution: RUBRICFORGE

The authors, Darragh Quinn and their team, wanted to fix this. They didn't want to just write a better rulebook by hand (which is slow and might miss things) or retrain the Judge's brain (which is expensive and creates a "black box" we can't understand). Instead, they invented RUBRICFORGE.

Here is how it works, step-by-step:

The Training Ground: They take a small set of real agent attempts where they already know the true outcome (the "ground truth"). Some succeeded, some failed.
The Evolution: They use a process called "reflective evolution." Imagine a robot trying to write a rulebook. It writes a draft, tests it on the known cases, and sees where it made mistakes. Then, it looks at the mistakes and asks, "Why did I get this wrong? What was the real clue I missed?" It rewrites the rulebook to fix that specific error. It does this over and over, getting smarter with every try.
The Freeze: Once the rulebook is perfect for the training cases, they "freeze" it. They don't change the Judge's brain at all; they just give it this new, super-specific set of instructions.
The Test: They use this frozen Judge on new, unseen agent attempts. Because the Judge is following a text-based rulebook, every time it gives a grade, it can point to exactly which rule it used. It's transparent.

What They Found: The "Harder to Fool" Discovery

The results were fascinating and a bit counter-intuitive. The authors were very honest about what they found and what they didn't.

First, they checked if the new Judge agreed with the "truth" more often than a standard, generic Judge. The answer was: not really. On a simple pass/fail test, the new method and the old method were basically tied. The paper explicitly states that the difference in overall agreement was not statistically significant. In other words, if you just asked, "Did they agree with the answer key?", the new method didn't win a clear victory.

However, the real win was in safety. The paper focuses on a specific type of error: the False Pass. This is when the Judge says "Success" to an agent that actually failed. This is the dangerous error because it ships broken software.

The standard generic Judge gave a "Pass" to failed agents about 17.3% of the time (on the $\tau$ -bench dataset).
The new RUBRICFORGE Judge only gave a "Pass" to failed agents about 11.5% of the time.

That's a huge drop. The new method caught three specific cases where the old Judge was fooled by a smooth story, and the new Judge correctly said "Fail." Crucially, the new Judge never made a mistake the old one didn't already make; it only caught more errors. It is "harder to fool."

The "Fluent Failure" Trap

The paper highlights that this advantage shows up most when the agent is "fluent"—when it talks a lot and sounds very confident.

In the Airline domain (a hard task), the old Judge was fooled 43.8% of the time on failed attempts.
The new Judge was fooled only 25.0% of the time.
This was especially true for agents using a "self-evolving" strategy that wrote long, detailed reports. The old Judge loved the long reports; the new Judge looked at the evidence and saw the failure.

The paper also looked at "graded" scores (like a 0 to 100 grade). Here, the new method was better at ranking (knowing which agent was better than the other), but the old method was slightly better at giving the exact right number. The authors are careful to say: if you need to compare agents to see who is best, use the new method. If you need a precise score for a report, the old method might be slightly more accurate.

Why This Matters

The most important takeaway is a shift in how we should measure success. The paper argues that for AI agents, agreement is not the goal; faithfulness is. A judge that agrees with the truth 90% of the time but misses the 10% of broken agents that look perfect is useless. A judge that catches those broken agents, even if it's slightly less perfect at other things, is safe to use.

By grounding the Judge's rules in real, labeled examples of failure, the authors created a system that resists the "Goodhart's Law" trap (where optimizing for a metric makes the metric useless). They didn't just make the Judge smarter; they made it more honest. The final verdict? The new method is harder to fool than it is to agree with, and that is exactly what we need to trust our AI agents in the real world.

Technical Summary: Inducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent Evaluation

1. Problem Statement

Evaluating language-model agents at scale increasingly relies on "LLM-as-a-judge" systems because the ground-truth signal (an executable environment reward) is often expensive, slow, or destructive to query during deployment. Current automatic judges typically rely on either hand-written scoring rubrics (e.g., G-EVAL) or fine-tuned model weights.

The paper identifies a critical, asymmetric failure mode in these existing judges: over-crediting. Judges tend to reward fluent, verbose, and confident narratives even when the underlying agent trajectory fails to achieve the true goal (e.g., a booking confirmation that narrates success but never commits the transaction).

False Pass (Over-crediting): Scoring a failed trajectory as a success. This is dangerous as it silently certifies broken agents, inflates success rates, and masks regressions.
False Fail: Scoring a successful trajectory as a failure. This is merely inefficient, costing a retry or a discarded sample.

The authors argue that for a reward-free evaluator to be safe for deployment, the primary metric of success should not be aggregate agreement with the ground truth, but rather the minimization of the false-pass rate.

2. Methodology: RUBRICFORGE

The paper proposes RUBRICFORGE, a method that induces the text of an agent-judging rubric rather than writing it by hand or fine-tuning model weights. The core innovation is optimizing the rubric string itself while keeping the judge model frozen.

Core Components

Frozen Backbone: A single language model (Qwen2.5-7B) serves as both the agent and the judge. The model weights are never updated; only the rubric text is optimized.
Reflective Evolution: The rubric is evolved using GEPA (a reflective prompt evolution algorithm).
- Input: A small set of ground-truth-labeled trajectories (training pool).
- Process: The current rubric scores the training trajectories. Mis-scored cases (where the judge's verdict disagrees with the ground-truth label) are collected.
- Reflection: A meta-prompt presents the current rubric, the mis-scored cases, and their true labels to the model. The model is instructed to propose a revised rubric that correctly identifies the observable evidence of failure (e.g., "tool errors," "missing confirmation") rather than simply memorizing the label.
- Selection: The evolved rubrics are evaluated on a held-out validation set. The best-performing rubric is selected.
Freezing and Application: The selected rubric is frozen verbatim. At test time, it is applied to held-out trajectories in a single model call with no environment access.

Key Design Choices

Grounding: The induction is grounded in external, programmatic outcome labels, preventing the rubric from drifting toward the judge's internal biases (fluency).
Interpretability: Since the optimized artifact is human-readable text, every verdict can be attributed to specific, named criteria (e.g., "Successful Authentication," "Order Verification").
Safety: The output contract is strict JSON; any parsing failure defaults to a "fail" verdict, ensuring the system is conservative (safe direction) rather than spuriously passing broken agents.

3. Key Contributions

RUBRICFORGE: An automatic metric induction method that evolves a judge rubric against ground-truth labels using a frozen backbone, producing concrete, human-readable, trajectory-grounded criteria.
Faithfulness over Agreement: The paper demonstrates that induction significantly improves faithfulness (reducing false passes) even when it does not significantly improve raw binary agreement.
Improved Ranking: On graded tasks, the induced rubric ranks outcomes more faithfully (higher Spearman correlation) than generic judges, despite being marginally worse in absolute score calibration.
Evaluation Protocol: The authors propose a comprehensive protocol for reward-free evaluation that includes:
- An oracle-reward ceiling.
- A specific false-pass rate probe.
- A split between ranking faithfulness and absolute calibration.
- Per-criterion leave-one-out attribution.
- Stratification by difficulty and fluency.

4. Experimental Results

Experiments were conducted on $\tau$ -Bench (retail/airline customer service, binary reward) and WEBSHOP (shopping, graded reward).

$\tau$ -Bench (Binary Success)

Agreement: RUBRICFORGE achieved 0.774 accuracy vs. 0.726 for G-EVAL. However, this difference was not statistically significant (McNemar $p = 0.248$ ). The authors explicitly state that on hard binary labels, the frozen 7B judge is near its agreement ceiling.
Over-Crediting (False-Positive Rate): This is where the method excels.
- G-EVAL false-pass rate: 0.173 (9/52 failures).
- RUBRICFORGE false-pass rate: 0.115 (6/52 failures).
- Result: RUBRICFORGE over-credits roughly half as often. The entire margin between the two judges consists of three trajectories where G-EVAL failed (over-credited) and RUBRICFORGE correctly failed. There were zero reversals (RUBRICFORGE never passed a trajectory G-EVAL failed).
Case Studies: The three caught trajectories were fluent but failed due to specific observable errors (e.g., "order not found," "payment mismatch"). RUBRICFORGE correctly identified these via its induced criteria (e.g., "Order Verification," "Request Fulfillment"), whereas G-EVAL was swayed by the confident final narrative.

WEBSHOP (Graded Reward)

Ranking: RUBRICFORGE achieved a Spearman correlation of 0.410 vs. G-EVAL's 0.370, indicating better faithfulness in ordering outcomes.
Calibration: G-EVAL was marginally better at absolute score calibration (Mean Absolute Error difference of -0.048, $p = 2 \times 10^{-4}$ ). The authors note that RUBRICFORGE's scores tend to be more decisive (0.2 or 1.0), which sharpens ranking but inflates absolute error against smooth graded rewards.

Stratification

The advantage of RUBRICFORGE was not uniform. It concentrated in harder domains (Airline vs. Retail) and fluent policies (RSEA vs. ReAct). The method was most effective at catching "fluent-but-wrong" trajectories, which are the primary failure mode of generic judges.

5. Significance and Claims

The paper's central claim is modest but precise: Induction makes a reward-free evaluator "harder to fool than to agree."

Deployment Relevance: For a reward-free evaluator, the false-pass rate is the critical deployment metric. A false pass ships a broken agent; a false fail merely costs a retry. RUBRICFORGE reduces this dangerous error type significantly without degrading performance on other metrics.
Grounding vs. Goodhart's Law: The authors connect their method to Goodhart's law. An ungrounded judge optimizes for "looking successful" (fluency), leading to metric drift. By grounding the rubric induction in checkable, external labels, RUBRICFORGE resists this drift, conferring faithfulness.
Interpretability: Unlike black-box fine-tuning, the induced rubric is text. Practitioners can inspect, edit, and audit the specific criteria (e.g., "Successful Authentication") that drive a verdict, making the evaluation process transparent.

The authors conclude that while aggregate agreement is often the headline metric, it is the wrong headline for reward-free evaluation. The true value of RUBRICFORGE lies in its ability to reduce the asymmetric risk of shipping broken agents, achieved by grounding the evaluation objective in true outcomes rather than the judge's untethered intuitions.