ExecRubrics: Executable Tool-Augmented Rubrics for Verifiable and Efficient Long-Form Evaluation

In the rapidly evolving world of artificial intelligence, large language models have become powerful tools for writing, reasoning, and solving problems. Yet, as these systems grow more capable, a significant challenge has emerged: how do we know if their answers are truly good? Currently, many developers rely on automated systems that act as judges, where one artificial intelligence evaluates the work of another. While this method is fast, it often operates like a black box. The judge might give a high score to a response without explaining why, or it might miss subtle errors because the criteria for "goodness" are vague. This lack of transparency is dangerous in high-stakes fields like healthcare or finance, where a single misunderstood instruction could lead to serious consequences. To solve this, researchers have long tried to use rubrics—detailed checklists of criteria that break down a complex task into smaller, manageable parts. However, traditional rubrics are written in natural language, which leaves them open to interpretation. A phrase like "be helpful" or "avoid medical errors" can mean different things to different judges, reintroducing the very subjectivity the system was meant to eliminate.

A team of researchers at Emory University and the University of Waterloo has proposed a new way forward called ExecRubrics. Instead of leaving evaluation criteria as vague sentences, they translate them into compact, executable computer programs. Imagine a rubric not as a paragraph of text that a human or a machine has to read and interpret, but as a precise set of instructions that a computer can run directly. In this framework, the criteria for a good answer are written as code. This code can check for specific things, such as whether a response mentions a particular medical warning, follows a specific format, or avoids certain types of logical errors. Because the logic is written in code, it is deterministic, meaning it will always produce the same result for the same input, removing the guesswork and variability found in human or black-box judging. The researchers tested this idea on three different benchmarks, including a dataset of medical advice and another focused on the quality of arguments. They found that these executable rubrics could rank responses just as well as, and sometimes better than, traditional language-based rubrics that rely on expensive AI judges. More importantly, the new method was dramatically faster, running up to 320 times quicker than the standard approach, while providing a clear, inspectable record of exactly why a response received a certain score.

The core of this work lies in transforming the intent of a rubric from a description into an operation. In a standard setup, a rubric might say, "The response should appropriately recommend urgent care." To evaluate this, a separate AI judge must read the response and decide if it meets that standard. This decision is subjective and can vary depending on which AI model is doing the judging. The ExecRubrics approach replaces this step. The researchers used advanced language models to convert that natural-language instruction into a Python program. This program doesn't just read the text; it actively scans it for specific patterns. It might look for keywords related to emergencies, check if the response suggests calling a doctor, or verify that it avoids giving false reassurance for serious symptoms. If the response contains the right elements, the code awards points; if it misses them or includes dangerous advice, it deducts points. This process is not a simulation or a guess; it is a direct execution of rules. The researchers showed that by allowing these programs to use standard text-processing tools—libraries that can count words, identify names, or detect negation—they could create even more sophisticated evaluators. These tools act like a magnifying glass for the code, allowing it to spot details that a simple text search might miss, such as whether a medical response correctly handles a patient's allergy or whether an argument acknowledges a counterpoint.

When the team applied this method to real-world datasets, the results were striking. On a benchmark for medical advice called HealthBench, the executable rubrics achieved a preference accuracy of 53 percent, matching the performance of the best natural-language rubric baselines. On a dataset for argument quality called ArgQuality, the executable rubrics reached 92 percent accuracy, outperforming the traditional methods. In a third test on a general helpfulness dataset called HelpSteer, they achieved 78 percent accuracy. These numbers indicate that the code-based evaluators were successfully distinguishing between high-quality and low-quality responses without needing a human or a separate AI to interpret the rules every time. The speed of the system was equally impressive. While the traditional method of using an AI judge to evaluate each criterion took significant time, the executable rubrics completed the same tasks in a fraction of a second. In some cases, the new method was 320 times faster than the old way. This speed, combined with the ability to see exactly which rule was triggered, offers a powerful alternative for domains where precision and speed are critical.

The researchers also explored how these rubrics handle complex logic that goes beyond simple checklists. In a traditional rubric, criteria are often treated as independent items that are simply added together. However, real-world evaluation often requires more nuance, such as handling exceptions or applying penalties. The executable rubrics could encode these dependencies directly. For instance, in a medical query about a penicillin allergy, the code could activate a specific branch that rewards the suggestion of non-penicillin alternatives while simultaneously penalizing any recommendation of amoxicillin. This kind of conditional logic is difficult to capture in a static checklist but is natural for a computer program. The study demonstrated that by incorporating external tools, the rubrics could become even more expressive, checking for things like readability, emotional tone, or the presence of specific safety warnings. This flexibility suggests that the approach can adapt to the unique needs of different domains, from healthcare to legal reasoning, without losing its transparency.

Despite these successes, the authors are careful to note that this is not a perfect solution. They point out that while the scoring logic is now explicit, the rubrics themselves are generated by AI, which means they could potentially be flawed or biased if the initial instructions are unclear. There is also a risk that if these rubrics are used to train AI models, the models might learn to "game" the system by including specific keywords or structures that trigger a high score without actually providing a good answer. The researchers emphasize that executable rubrics should be viewed as a tool for making evaluation more transparent and accountable, not as a replacement for human oversight. In high-stakes environments, human experts are still needed to review the rules, ensure they are fair, and validate that the code aligns with real-world needs. The study suggests that this approach is a significant step toward a future where AI evaluation is not just a black box, but a clear, auditable process that can be inspected, edited, and trusted.

The implications of this work extend beyond just making evaluation faster. By turning evaluation criteria into code, the researchers have created a bridge between the abstract goals of human judgment and the precise execution of computer programs. This allows for a level of consistency that is difficult to achieve with human raters or opaque AI judges. If a rubric needs to be changed, it can be edited directly in the code, and the impact of that change can be immediately tested. This makes the evaluation process more dynamic and responsive to new requirements. The study also highlights the potential for these rubrics to be used in training AI models, providing a clear signal of what constitutes a good response. However, the authors caution that the success of the system depends heavily on the quality of the generated code and the tools available to it. In some cases, the executable rubrics performed slightly worse than the best human-written baselines, particularly in complex medical scenarios where the nuance of a situation might be hard to capture in code alone. This suggests that while the method is powerful, it is not yet a complete replacement for human expertise.

Ultimately, the research offers a new perspective on how we can evaluate the quality of artificial intelligence. It moves away from the idea of a single, mysterious score and toward a system of transparent, verifiable rules. The findings suggest that by making the logic of evaluation explicit, we can build systems that are not only more efficient but also more trustworthy. As AI continues to integrate into critical aspects of society, the ability to audit and understand how decisions are made becomes increasingly important. ExecRubrics provides a framework for doing just that, turning the abstract art of grading into a concrete, executable science. The work does not claim to have solved all the problems of AI evaluation, but it offers a promising path forward, one where the criteria for success are as clear and reliable as the code that enforces them.