🤖 machine learning

Codifying the Judge: Scalable Evaluation via Program Distillation

The paper introduces PAJAMA, a scalable evaluation system that distills LLM decision logic into transparent, cost-effective programmatic judges to match the performance of large language model judges while significantly reducing latency and API costs.

Original authors: Tzu-Heng Huang, Shengqi Qiu, Frederic Sala

Published 2026-07-28

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Tzu-Heng Huang, Shengqi Qiu, Frederic Sala

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the head of a massive library, and every day, thousands of people submit stories they wrote. Your job is to read them, decide which ones are the best, and give them awards. In the world of artificial intelligence, this is exactly what happens when computers try to judge other computers. We call this "LLM-as-a-judge," where a super-smart AI reads two answers and picks the winner. It's become the standard way to test how good AI models are. But there's a huge problem: asking a super-smart AI to read every single story is incredibly expensive, slow, and sometimes the AI makes up reasons for its choices that don't actually make sense. It's like hiring a famous, expensive critic to read every single book in the library; eventually, you run out of money, and you can't even tell if the critic is being fair or just having a bad day.

This is where the new paper comes in. The researchers asked a simple question: instead of hiring the famous critic to read every single book, what if we asked the critic to write a rulebook once, and then let a team of fast, cheap robots read the books based on those rules? They call this "program distillation." Instead of paying for the expensive AI to do the thinking every time, they teach it to write a set of instructions (a computer program) that can do the judging instantly. It's like turning a slow, expensive human chef into a fast, automated toaster that makes perfect toast every time, as long as you give it the right recipe. The paper shows that this new method is not only much faster and cheaper but also more honest about how it makes decisions, because you can actually read the code to see exactly why it picked a winner.

The Paper's Big Idea: Turning a Brain into a Toolbox

The authors of this paper, Tzu-Heng Huang, Shengqi Qiu, and Frederic Sala from the University of Wisconsin-Madison, are tackling the "expensive and opaque" problem of AI judging. They propose a system called PAJAMA (Program-As-a-Judge Automated Model Assessment). Think of PAJAMA not as a single judge, but as a workshop where they build a whole team of specialized tools to do the grading.

Here is how the magic happens, step by step:

1. The One-Time "Brain Dump"
Usually, when you want an AI to judge a pair of answers, you have to send the question and the answers to the AI every single time. This costs money and takes time. PAJAMA changes the game. Instead of asking the AI to judge every answer, they ask the AI to write a program (a piece of code) that knows how to judge.
Imagine you have a brilliant but slow art critic. Instead of asking them to look at 10,000 paintings one by one, you ask them to write a manual for a robot. The manual says things like, "If the painting has a blue sky, give it 5 points. If the sky is messy, subtract 2 points." Once the critic writes this manual, you don't need them anymore. You just run the robot. The paper shows that the AI can write these "judging manuals" (programs) that are surprisingly good at spotting logical errors, checking if the story makes sense, or seeing if the answer is too short.

2. The Committee of Robots
One robot might be bad at math but great at grammar. Another might be great at spotting lies but bad at style. So, PAJAMA doesn't just make one program; it makes a whole committee of them. They ask the AI to write many different programs, each looking at the answer from a slightly different angle.
To make sure these robots don't all say the same thing (which would be boring and useless), the researchers give them different "rubrics" or checklists. One robot checks for logical flow, another checks for topic consistency, and another checks for formatting. It's like having a panel of judges where one is a grammar teacher, one is a logic professor, and one is a style coach.

3. The "Confidence" Switch
Sometimes, even a committee of robots gets confused. Maybe the two answers are so similar that the robots can't decide. This is where PAJAMA gets clever. It has a "confidence meter." If the robots are sure, they shout out the winner. But if they are unsure, or if they all abstain (say, "I don't know"), PAJAMA has a backup plan. It sends those tricky cases to the expensive, super-smart AI judge (the "LLM") just for those specific questions.
This is like a security guard at a museum. If a visitor looks normal, the guard lets them pass instantly. But if someone looks suspicious, the guard calls the manager to take a closer look. This way, you only pay the expensive manager for the hard cases, saving a ton of money on the easy ones.

What They Found: Fast, Cheap, and Surprisingly Smart

The researchers tested this system on five different datasets and with four different families of AI models. Here is what they discovered:

Speed is King: The programmatic judges are incredibly fast. They can process answers 47.25 times faster than a standard AI judge. While a regular AI judge might take a second or two to think, these programs do it in a flash because they are just running simple math and logic rules, not trying to "think" like a human.
Accuracy is High: Even though they are fast and cheap, they are still very good. The team of programs matched the accuracy of a large, 13-billion-parameter AI model (a very smart model) on average. In some cases, a small team of just 8 programs was as accurate as a 7-billion-parameter model.
The "Pareto Frontier" Breakthrough: In science, the "Pareto frontier" is the best possible balance between two things, like speed and accuracy. Usually, if you want more speed, you lose accuracy. PAJAMA breaks this rule. By using the program to handle the easy stuff and the AI to handle the hard stuff, they found a sweet spot where they got both higher accuracy and higher speed. For example, when paired with a specific AI model, they improved accuracy by 5% while being nearly 3 times faster.
Cheaper Training: They also used these programs to train other AI models (a process called "reward model distillation"). They found that using the programs' judgments to teach a new AI was 50 times cheaper than using a famous, expensive AI to do the teaching. The new AI trained on the cheap program data performed just as well, or even better, than the one trained on the expensive data.
Honesty and Bias: One of the biggest problems with AI judges is that they can be biased. They might like long answers just because they are long, or they might prefer answers with fancy formatting. Because PAJAMA's judges are actual computer programs, you can see exactly what they are doing. If a program is biased toward long answers, you can just open the code, find the line that says "add points for length," and delete it. The paper showed that by "calibrating" (fixing) the programs, they could significantly reduce these biases, making the judging much fairer.

Why This Matters

The paper argues that we don't need to keep paying huge bills to have AI judge AI. By turning the "thinking" part of the judge into a set of reusable, transparent rules, we can make evaluation scalable. It turns a black box (where we don't know why the AI made a choice) into a clear, inspectable process.

The authors are careful to note that this isn't a magic wand that replaces all AI judges. Sometimes, the problems are too complex for simple rules, and that's why they keep the expensive AI as a backup for the tough cases. But for the vast majority of everyday judging tasks, this "program distillation" approach offers a way to get high-quality results without the high cost, high latency, and hidden biases of the current methods. It's a shift from asking a genius to do all the work, to teaching the genius how to build a team of efficient workers.

Technical Summary: Codifying the Judge: Scalable Evaluation via Program Distillation

Problem Statement

The "LLM-as-a-judge" paradigm has become the standard for automated evaluation in machine learning pipelines, handling tasks such as pairwise preference labeling, reward model distillation, and reinforcement learning feedback. However, this approach faces four critical limitations that hinder scalability and reliability:

Inference Cost: Scaling evaluation to millions of samples using proprietary models (e.g., GPT-5, Gemini-3) incurs prohibitive API costs. Open-weight models reduce API bills but remain expensive regarding latency and GPU demand.
Opaque Logic: LLM decision processes are internal and opaque, making it difficult to verify if a verdict relies on stated rationale or hallucination.
Systemic Bias: LLM judges are sensitive to stylistic biases, often favoring verbosity, rich formatting, or emotionally charged language over factual accuracy.
Re-inference Tax: Current prompting pipelines are inflexible; revising a single evaluation criterion requires re-running inference over the entire dataset, incurring redundant costs.

Methodology: The PAJAMA Framework

The authors propose PAJAMA (Program-As-a-Judge Automated Model Assessment), a system that shifts from model-based inference to synthesized program execution. Instead of prompting an LLM for every sample, PAJAMA distills the LLM's decision logic into a committee of executable Python programs.

The framework operates through four main stages:

1. Distilling into Programmatic Judges

Rather than asking an LLM to assess each candidate repeatedly, PAJAMA asks the LLM once at synthesis time to generate the logic it would apply.

Synthesis: An LLM is prompted with curated evaluation rubrics (e.g., logical coherence, topic consistency) to generate Python functions (judging_function). These functions take query and response pairs as inputs and return scalar scores based on interpretable features (e.g., TF-vector cosine similarity, semantic drift).
Diversity Control: To prevent repetitive logic, the system curates a list of ten distinct rubrics and applies text-based similarity checks to filter out programs with overlapping logic, ensuring a diverse pool of judges.

2. Modeling Programmatic Judges

Since individual programs may be noisy or operate on different scales, PAJAMA employs a three-step modeling procedure:

Calibration: Program outputs are normalized to $[0, 1]$ . A per-program threshold ( $\tau_j$ ) is determined using a held-out validation set to convert score differences into discrete verdicts ( $+1$ , $-1$, or abstain $0$). This allows programs to withhold votes when their signal is weak.
Selection: Programs scoring below random chance on the validation set are discarded. The top- $k$ programs by validation accuracy are selected to form the final committee.
Aggregation: Using techniques from weak supervision (specifically label models like Snorkel), the system aggregates the noisy votes of the top- $k$ programs. A generative model estimates each program's accuracy based on agreement/disagreement patterns to produce a weighted joint verdict.

3. Routing Uncertain Cases (Hybrid Evaluation)

To handle inputs where the program committee abstains or lacks confidence, PAJAMA implements a fallback mechanism:

Routing Signals: Internal signals derived from the program committee—specifically vote variance (disagreement among programs) and aggregator posterior (confidence probability)—serve as routing indicators.
Fallback: Samples flagged as low-confidence are escalated to an LLM judge. This creates a hybrid system that maintains high throughput for clear-cut cases while ensuring accuracy for difficult ones.

4. Reward Model Distillation

The system also utilizes programmatic verdicts as training signals for reward models. By distilling labels from the program committee, PAJAMA generates a dataset for fine-tuning smaller models, avoiding the need for continuous proprietary LLM inference during the training phase.

Key Results

The authors evaluated PAJAMA across five preference datasets (JudgeLM, PandaLM, MultiPref, Prometheus, Preference-700K) and four model families (OLMo-2, Gemma-3, Qwen2.5, and proprietary judges).

Accuracy vs. Throughput: Standalone programmatic judges matched the accuracy of mid-sized LLM judges (e.g., OLMo-2-13B-INSTRUCT) while running 47.25× faster. On average, they achieved 78.11% accuracy, comparable to 13B parameter models, but with throughput orders of magnitude higher.
Pareto Frontier Advancement: When combined with LLM fallbacks, PAJAMA advanced the accuracy-throughput Pareto frontier. For instance, pairing with OLMo-2-7B-INSTRUCT improved accuracy by +5.0% at 2.9× the throughput compared to LLM-only evaluation.
Cost-Effective Distillation: On RewardBench, a reward model distilled from programmatic labels outperformed one trained on proprietary GPT-4 labels while reducing API costs by 50×. The cost structure shifts from linear ( $O(n)$ ) with dataset size to a one-time investment ( $O(1)$ ) for program synthesis.
Robustness to Bias: Programmatic judges demonstrated superior resilience to systemic biases (position, rich-content, reference, gender, verbosity) compared to LLM judges. Notably, position bias was eliminated (0.00% flip rate) because program logic is invariant to candidate order.
Calibratability: The system allows for targeted bias reduction via code edits. Using a coding agent to calibrate programs reduced bias metrics further (e.g., a 9.39% reduction in flip rate for rich-content bias), demonstrating that bias can be "patched" in the code rather than being an inherent, unfixable property of the model.

Significance and Contributions

The paper claims to introduce a new evaluation paradigm that replaces per-sample model inference with one-time program synthesis and local execution. The primary contributions are:

A New Evaluation Paradigm: Distilling LLM judging logic into executable programs offers a transparent, low-cost, and flexible alternative to black-box model assessment.
The PAJAMA System: A hybrid framework that synthesizes diverse judges, aggregates their verdicts via weak supervision, and employs a confidence-aware router to escalate uncertain cases, balancing speed and accuracy.
Advancing the Pareto Frontier: PAJAMA achieves competitive accuracy with mid-sized LLMs at a fraction of the cost and significantly higher throughput, pushing the boundaries of what is feasible for large-scale automated evaluation.
Cheap, High-Quality Reward Signals: The system provides a cost-effective method for generating training data for reward models, outperforming proprietary supervision at a fraction of the cost and enabling robust, bias-correctable evaluation pipelines.

The authors emphasize that this approach transforms evaluation from an opaque, expensive process into a transparent, auditable, and programmable workflow, addressing the fundamental scalability and reliability issues of current LLM-as-a-judge systems.