Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed as "judges" to evaluate code artifacts (e.g., code generation, repair, and test generation) in agentic software engineering workflows, especially when human review or executable test coverage is unavailable. However, the reliability of these judges is compromised by two critical, under-explored issues:

• Prompt-Induced Bias: LLM judges are highly sensitive to superficial, non-semantic cues in the prompt (e.g., position order, verbosity, authority cues, chain-of-thought). These cues can systematically shift verdicts even when the underlying code semantics remain unchanged.
• Inconsistency: Repeated evaluations of the same code pair under identical rubrics often yield different outcomes, undermining the reproducibility of model rankings and task-level conclusions.

The paper argues that current LLM-as-a-Judge studies often report accuracy without accounting for these biases, leading to fragile and potentially invalid evaluations of software engineering systems.

2. Methodology

The authors conducted a measurement-first study using a pairwise comparison framework across three distinct software engineering tasks:

1. Code Generation (CodeGen)
2. Code Repair (CodeRepair)
3. Unit Test Generation (TestGen)

Experimental Setup:

• Dataset: CodeJudgeBench, containing 5,352 curated "Good/Bad" response pairs across varying difficulty levels (Easy, Medium, Hard).
• Models Evaluated:
  • Qwen3-4B: A general-purpose open-source model.
  • Qwen2.5-Coder-3B: A code-specialized open-source model.
  • GPT (Closed Source): A proprietary API-hosted model (treated as a black box).
• Bias Suite (RQ1): The authors designed 12 explicit prompt-injected biases tailored to code evaluation, including:
  • Position/Order: Swapping candidate A and B.
  • Authority: Citing standards (e.g., PEP 8) for one candidate.
  • Chain-of-Thought (CoT): Providing reasoning steps for only one candidate.
  • Verbosity: Adding excessive comments/length to one candidate.
  • Refinement/Sentiment/Model-Name: Adding meta-text suggesting one candidate is "refined," "confident," or from a "trusted source."
  • Distraction/Bandwagon/Diversity: Injecting irrelevant social or stylistic cues.
• Metrics:
  • Accuracy: Micro-averaged accuracy of selecting the gold-standard answer.
  • Robustness Rate: How much accuracy shifts when a specific bias is introduced.
  • Consistency Rate (CR): The percentage of times the judge makes the same decision when the exact same prompt is run twice (test-retest reliability).
  • Answer Rate: The percentage of runs where the model outputs a valid structured verdict (A vs. B) rather than free-form text.

3. Key Contributions

1. Systematic Bias Audit: The first comprehensive study quantifying how 12 specific prompt biases affect LLM judges across multiple code tasks and model families.
2. Position Sensitivity Discovery: Demonstrated that LLM judges have a strong inherent preference for the first position (A) or second position (B) depending on the model, and that biases often amplify this positional bias rather than improving reasoning.
3. Consistency vs. Correctness Paradox: Revealed that certain biases (e.g., "Sentiment" or "Refined") can artificially inflate the Consistency Rate (making the model repeat the same wrong answer) while destroying accuracy.
4. Answer Rate Failure Mode: Identified that generic models (Qwen3-4B) frequently fail to emit structured verdicts (44% answer rate), whereas code-specialized models (Qwen2.5-Coder-3B) maintain high structural compliance (99%).
5. Methodological Guidelines: Proposed a new standard for SE evaluation requiring the reporting of bias sensitivity and consistency alongside accuracy, including A/B swapping and controlled perturbations.

4. Key Results

A. Effect of Bias on Accuracy (RQ1)

• Directional Reversal: The impact of a bias depends entirely on the position of the correct answer.
  • Example: Chain-of-Thought (CoT) and Sentiment significantly boost accuracy when the correct answer is in Position A (e.g., +9% on CodeRepair) but cause catastrophic drops when the correct answer is in Position B (e.g., -12% on CodeRepair).
  • Example: Verbosity often suppresses the model's preference for Position A, thereby improving accuracy when the correct answer is in Position B.
• Task Sensitivity:
  • CodeRepair: Generally the most robust task but still susceptible to large swings (e.g., Refined bias changes accuracy from ~74% to ~84% or drops to ~67% depending on position).
  • TestGen: The most fragile task. Biases like Distraction and Verbosity caused accuracy to drop from ~77% to ~62% on GPT, and showed massive variance across models.
• Model Differences:
  • Qwen2.5-Coder-3B: Highly sensitive to position. Without bias, it struggles when the correct answer is in Position B (accuracy <50%). Biases like "Refined" or "Sentiment" create near-ceiling performance on A but near-floor performance on B.
  • GPT: Shows a strong baseline (~80% accuracy) but is still vulnerable. TestGen is its weakest point, where biases cause significant degradation.

B. Consistency and Reliability (RQ2)

• Baseline Instability: Qwen2.5-Coder-3B has a very low baseline Consistency Rate (CR) on TestGen (~50%, equivalent to random guessing).
• The "False Reliability" Trap: Introducing strong biases (e.g., Sentiment, Refined) can artificially inflate the Consistency Rate to >80% for Qwen2.5-Coder-3B. However, this high consistency often reflects the model blindly anchoring to the bias cue rather than evaluating code, leading to systematically wrong but consistent decisions.
• GPT Stability: GPT shows high baseline consistency (~85-92% for CodeGen/Repair) but still exhibits vulnerability on hard TestGen instances where biases can force a consistent but incorrect decision.

C. Answer Rate

• Qwen2.5-Coder-3B: ~99% answer rate (reliably outputs [[A>B]] or [[B>A]]).
• Qwen3-4B (Generic): ~44% answer rate. It frequently fails to follow the structured output schema, generating free-form text or solving the problem instead of judging, rendering its accuracy metrics on "answered" items misleading for large-scale evaluation.

5. Significance and Implications

• Validity of SE Research: Reported improvements in code generation or repair models based on LLM judges may be artifacts of prompt design (e.g., favoring a specific position or verbosity) rather than genuine capability.
• Agentic Workflows: In autonomous agent loops, a biased judge can steer the system toward suboptimal patches or incorrect test cases, creating a feedback loop of error.
• New Evaluation Standards: The paper argues that "Accuracy" is insufficient. Future studies must report:
  • Bias Sensitivity: How much does accuracy change with prompt perturbations?
  • Consistency Rate: How stable are the decisions across runs?
  • Answer Rate: How often does the judge actually produce a valid verdict?
• Practical Mitigation: Researchers should implement A/B order swapping, controlled prompt variations, and multi-run aggregation to ensure conclusions are robust to presentation artifacts.

In conclusion, the paper demonstrates that LLM-as-a-Judge is not a "drop-in" replacement for human evaluation. Its decisions are fragile, highly dependent on prompt framing, and prone to systematic biases that can invalidate software engineering research if not rigorously controlled.