💬 NLP

Comparing Developer and LLM Biases in Code Evaluation

This paper introduces TRACE, a framework that reveals significant misalignment between LLM judges and human developers in code evaluation across three modalities, showing that even the best models underperform humans by 12–23% and exhibit systematic biases, such as favoring longer explanations, on most existing code quality dimensions.

Original authors: Aditya Mittal, Ryan Shar, Zichu Wu, Shyam Agarwal, Tongshuang Wu, Chris Donahue, Ameet Talwalkar, Wayne Chi, Valerie Chen

Published 2026-03-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aditya Mittal, Ryan Shar, Zichu Wu, Shyam Agarwal, Tongshuang Wu, Chris Donahue, Ameet Talwalkar, Wayne Chi, Valerie Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to teach a robot how to cook. You ask the robot to make a sandwich, and it gives you two options. You pick the one with the fresh, crisp lettuce. But the robot's "taste tester" (another AI) picks the one with the soggy, wilted lettuce because it thinks "more lettuce = better sandwich."

This is the core problem explored in the paper "Comparing Developer and LLM Biases in Code Evaluation."

Here is the breakdown of what the researchers found, using simple analogies:

1. The Problem: The Robot Judge is Out of Touch

As we use AI to write code, we need a way to grade that code. Instead of hiring a human to check every single line, we use a "Judge AI" to say, "This code is good, that code is bad."

The researchers asked: Do these Judge AIs actually think like human developers?

They found that no, they don't. Even the smartest AI judges are missing the mark. They are about 12% to 23% worse at picking what a human developer likes than a human picking what another human likes. It's like having a food critic who loves burnt toast because they think it looks "rustic," while everyone else just wants a fresh slice.

2. The Tool: TRACE (The "Rubric Detective")

To figure out why the robots are wrong, the authors built a tool called TRACE.

Think of TRACE as a detective's magnifying glass.

Step 1: It looks at a coding task where a human picked "Option A" and the AI Judge picked "Option B."
Step 2: It breaks down the two options into a checklist of "Rubric Items" (like a scorecard). These items might be things like "Is the code fast?", "Are there comments?", or "Is it easy to read?"
Step 3: It compares the scorecards. It asks: "Did the human care more about speed? Did the AI care more about comments?"

This revealed 35 specific ways that AI judges and human developers disagree.

3. The Three Scenarios (The "Kitchen" Settings)

The researchers tested the AI judges in three different "kitchens" (coding scenarios), and the robots got confused in different ways in each:

A. The Autocomplete Chef (Code Completion)

The Scene: You are typing code, and the AI suggests the next few words.
The Human Preference: Humans want the suggestion to be clear and easy to read within the context of the file they are already working on.
The Robot Bias: The AI judges loved suggestions that were functionally correct but messy or hard to read. They ignored the fact that the code needed to fit smoothly into the existing file.
Analogy: The human wants a sandwich that fits perfectly on their plate. The robot judge picks the sandwich that is technically edible but is 3 feet tall and spills everywhere.

B. The Editor (Code Editing)

The Scene: You highlight a broken piece of code and tell the AI, "Fix this."
The Human Preference: Humans want the fix to be precise and clear. They don't want the AI to rewrite the whole file or add unnecessary fluff.
The Robot Bias: The AI judges often preferred answers that were vague or added too much "explanation" text, rather than a clean, surgical fix.
Analogy: You ask the robot to replace a broken lightbulb. The human wants a new bulb. The robot judge picks the answer where the robot replaced the bulb and rewired the whole house and wrote a 10-page manual, even though you just wanted the light on.

C. The Chatbot (Chat-based Coding)

The Scene: You ask the AI a question like, "How do I sort this list?"
The Human Preference: Humans want a smart, context-aware answer that shows it understands the specific problem.
The Robot Bias: The AI judges loved generic, long-winded explanations. They thought "more words = better answer," even if the answer was boring and didn't solve the specific problem.
Analogy: You ask, "How do I fix a flat tire?" The human wants the step-by-step instructions. The robot judge picks the answer that writes a 500-word essay on the history of rubber tires, because it sounds "impressive."

4. The Big Surprise: Even "Special" Robots Fail

The researchers tested 13 different AI models. Some were general-purpose (like a Swiss Army Knife), and some were "Specialized Judges" trained specifically to grade code.

The Result: The specialized judges didn't do much better than the general ones.

Analogy: It's like hiring a professional food critic to judge a burger. You'd expect them to be great. But in this case, the "professional" critic was just as confused as the "amateur" one. They both missed the subtle things that make a burger (or code) actually good for a human.

5. The Conclusion: We Need Better Training

The paper concludes that we can't just trust AI to grade AI yet. The "rubrics" (the rules they use to judge) are different from the rules humans use.

Humans care about: Readability, context, precision, and "feeling right."
AI Judges care about: Length, generic correctness, and surface-level features.

The Takeaway:
If we want AI to help us build software, we need to teach the "Judge AIs" to think more like human developers. We need to stop training them to just look for "correctness" and start training them to understand context, clarity, and the messy reality of how people actually work.

Until then, the AI Judge is like a robot that thinks a messy, over-complicated solution is a masterpiece, while the human developer just wants to get the job done.

1. Problem Statement

As Large Language Models (LLMs) are increasingly deployed as "judges" to evaluate code generation and editing tasks, there is a critical need to understand if they align with human developer preferences. Existing evaluation methods often rely on static, polished code from repositories or competitive programming benchmarks, which fail to capture the messy, ambiguous, and context-dependent nature of real-world software development.

The core problem addressed is the misalignment between LLM judges and human developers. Specifically:

Do LLM judges accurately predict what developers prefer in realistic interactive settings (IDE autocompletion, instructed edits, and chat)?
What specific biases do LLMs exhibit compared to humans when weighing code quality criteria (rubric items)?
Do current "judge" models (fine-tuned or specialized) outperform general-purpose models in capturing these nuances?

2. Methodology: The TRACE Framework

The authors propose TRACE (Tool for Rubric Analysis in Code Evaluation), a framework designed to evaluate LLM judges in realistic workflows and automatically extract the criteria (rubrics) driving their decisions. The methodology consists of three stages:

Stage 1: Measuring Alignment

Datasets: The framework evaluates judges across three interaction modalities:
1. Code Completion: In-file autocompletion (from Copilot Arena).
2. Instructed Code Edits: Modifying existing code based on instructions (from EDIT-Bench).
3. Chat-based Coding: Open-ended problem solving (filtered from Chatbot Arena).
Judges: 13 models were tested, including general-purpose models (e.g., GPT-5, Claude Sonnet 4), specialized judge models (e.g., Prometheus 2, Atla Selene), and reward models.
Metric: Accuracy in predicting human pairwise preferences, controlling for positional bias by swapping response orders.

Stage 2: Automatic Rubric Discovery

To explain why judges and humans disagree, TRACE automatically extracts evaluation criteria:

LLM-Generated: An LLM analyzes pairs of responses to identify concrete differences (e.g., "one handles errors better").
Human-Annotated: Human engineers provide justifications for their preferences, which are abstracted into general criteria.
Aggregation: These criteria are merged into a comprehensive, interpretable Rubric (a set of axes like "Error Handling," "Clarity," "Modularity").

Stage 3: Bias Quantification

Scoring: An LLM ranker scores each response pair against every rubric item (values: -1, 0, 1).
Modeling: Logistic regression models are trained separately for Human Preferences and LLM Judge Preferences using the rubric scores as features.
Comparison: The authors compare the regression coefficients ( $\beta$ ) of humans vs. judges. A significant difference indicates that the judge overweights or underweights a specific quality dimension compared to humans.

3. Key Contributions

TRACE Framework: A novel pipeline for not only measuring judge accuracy but also diagnosing the source of misalignment through automatic rubric extraction and coefficient analysis.
Multi-Modal Evaluation: The first comprehensive study comparing LLM judges against developers across three distinct, realistic coding modalities (completion, editing, chat), rather than static benchmarks.
Systematic Bias Identification: The discovery of 35 significant sources of misalignment between humans and judges, revealing that judges often prioritize different aspects of code quality than developers do.
Benchmarking Judge Models: A rigorous evaluation of 13 models, showing that specialized judge models do not consistently outperform general-purpose models in code evaluation.

4. Key Results

A. Performance Gap

LLMs Underperform Humans: Even the best LLM judges underperform human annotators by 12–23 percentage points in predicting human preferences across all modalities.
No Dominant Model: No single model consistently dominates. Fine-tuned judge models do not reliably outperform general-purpose instruction-tuned models, suggesting current training data or specialization strategies are insufficient.
Positional Bias: LLMs exhibit significant positional bias (preferring the first or second option regardless of content), whereas reward models are order-invariant.

B. Specific Misalignments (Rubric Analysis)

The study identified distinct bias patterns depending on the interaction modality:

Code Completion: Judges overweight "Functional and Logical Alignment" (does the code run?) but underweight "Explicitness and Clarity" (is it readable within the file context?). Humans prioritize readability and context integration.
Instructed Edits: Judges tend to treat edits as constraint satisfaction tasks, focusing on whether the change was applied. They underweight "Clarity" and "Structure," whereas humans view edits as opportunities to improve overall code quality and readability.
Chat-based Coding: Judges overweight generic "Code Explanations" and clarity of text, while humans prefer Domain-Specific Detail and context-aware solutions. Judges often reward verbose explanations that humans find unnecessary.

C. Alignment with Software Engineering Criteria

The extracted rubrics map to 16 themes, 11 of which align with established software engineering criteria (e.g., Syntax, Formatting, Robustness).
Despite this alignment, LLM judges remain significantly misaligned with human preferences on 6 out of these 11 themes. This suggests that even when judges evaluate "standard" criteria, their internal weighting differs fundamentally from human intuition.

5. Significance and Implications

Limitations of Current Benchmarks: The paper demonstrates that high performance on static benchmarks (like RewardBench or JudgeBench) does not translate to real-world developer preference alignment.
Need for Rubric-Aware Training: Simply fine-tuning models on preference data is insufficient. Future judge training must explicitly address the specific rubric dimensions where misalignment occurs (e.g., teaching models to value context-aware clarity over generic explanations).
Human-in-the-Loop Necessity: Given the 12-23% gap and the systematic nature of biases, LLM judges cannot yet fully replace human evaluation in critical software engineering workflows. They should be used as assistants with human oversight, particularly for tasks involving code readability and maintainability.
Methodological Advance: TRACE provides a scalable method for auditing AI evaluators, moving beyond "black box" accuracy metrics to interpretable, criteria-based diagnostics.

In conclusion, the paper argues that while LLMs are powerful tools, their current iteration as judges fails to capture the nuanced, context-dependent priorities of human developers, particularly regarding code readability, domain specificity, and the balance between functionality and maintainability.