Technical Summary: Judging Against the Reference

Problem Statement

The paper addresses a critical, under-explored failure mode in the "LLM-as-a-judge" paradigm for Question Answering (QA) evaluation. While Large Language Models (LLMs) are increasingly used to automatically grade candidate answers against a provided reference (gold) answer, their ability to strictly adhere to that reference when it conflicts with the model's internal parametric knowledge is poorly understood. The authors identify that when a provided reference answer is intentionally swapped with an incorrect entity that contradicts the judge's internal beliefs, the judge often disregards the provided reference. Instead of evaluating the candidate answer based on the given instruction and reference, the judge reverts to its own parametric knowledge, leading to unreliable scores and degraded evaluation fidelity. This issue is particularly salient because factual knowledge evolves, and datasets may intentionally encode counterfactual targets that diverge from a model's training data.

Methodology

To systematically investigate this phenomenon, the authors introduce a Swapped-Reference QA Evaluation Framework. This framework constructs controlled evaluation instances to induce conflicts between the provided reference and the judge's internal knowledge.

1. Dataset Construction: The authors curated four existing QA datasets (NaturalQuestions-Open, PopQA, SciQ, and FreshQA). They applied Named Entity Recognition (NER) to identify entities in reference answers and then performed Reference Swapping using several strategies:

   • Random Swap (RS): Replacing the original reference with another entity of the same type from the dataset (e.g., swapping one person for another).
   • Plausible Swap (PS): Replacing the reference with a plausible but incorrect answer generated by a strong model (GPT-5).
   • Popularity-High/Low Swap: Swapping references with entities of varying Wikipedia pageview popularity to test the effect of knowledge strength.
   • Evaluator-Knowledge Swap: Replacing the reference with the specific answer the judge model itself would generate, creating a scenario where the swapped reference aligns perfectly with the judge's internal belief.

2. Candidate Generation: For each swapped reference, the framework generates corresponding long-form candidate answers that align with the swapped reference, as well as candidates aligned with the original reference.

3. Meta-Evaluation: The study evaluates 13 different LLM judges (including GPT-4o, GPT-5, Llama-3, and Qwen-3 families) across these datasets. The core metric is Reference-Polarity Accuracy Gap (RPAG), defined as the difference in accuracy between the original reference setting ($ACC_o$) and the swapped reference setting ($ACC_s$). A high RPAG indicates that the judge's performance degrades significantly when the reference is swapped, revealing a failure to condition on the provided text.

4. Mitigation Analysis: The authors tested whether common prompting strategies—Direct instructions, In-Context Learning (ICL), Chain-of-Thought (CoT), and CoT with Self-Consistency—could mitigate this vulnerability.

Key Findings

• Widespread Vulnerability: Across all 13 judge models and four datasets, swapping the reference answer caused a sharp drop in grading reliability. Even strong models like GPT-4o and GPT-5 exhibited significant accuracy degradation (e.g., drops of 30–50% in Random Swap scenarios).
• Over-Reliance on Parametric Knowledge: The failure is driven by the judge's over-reliance on internal beliefs. When the swapped reference aligns with the judge's parametric knowledge (Evaluator-Knowledge Swap), the vulnerability disappears (RPAG approaches zero). Conversely, when the swapped reference conflicts with strong internal beliefs, the judge ignores the provided reference.
• Knowledge Strength and Popularity: Vulnerability correlates with the strength of the conflicting knowledge. Swaps involving highly popular entities (strong priors) cause greater accuracy drops than swaps involving obscure entities. Similarly, questions involving stable, long-standing facts show higher vulnerability than those involving fast-changing or time-sensitive knowledge.
• Scaling Does Not Help: Increasing model capacity (e.g., moving from 8B to 70B parameters) does not mitigate the issue; in some cases, larger models exhibited greater vulnerability, suggesting that increased capacity strengthens reliance on internal world knowledge rather than improving adherence to external instructions.
• Limitations of Prompting: Common mitigation strategies, including detailed instructions and Chain-of-Thought prompting, failed to resolve the issue. Notably, CoT prompting sometimes increased vulnerability under high-conflict (Random Swap) conditions, as the reasoning process appeared to amplify the model's reliance on its own parametric beliefs to override the provided reference.

Significance and Contributions

The paper makes three primary contributions:

1. Identification of a Critical Failure Mode: It uncovers that LLM judges frequently ignore evaluation instructions when the reference answer contradicts their parametric knowledge, a fundamental limitation in reference-conditioned evaluation.
2. Systematic Framework: It introduces a controlled swapped-reference QA evaluation framework that enables fine-grained analysis of judge adherence across datasets, swap types, and knowledge alignments.
3. Diagnostic Insights: It identifies that the root cause is an inability to resolve knowledge conflicts in favor of the provided reference, and that standard prompting techniques offer limited relief.

The authors conclude that these findings highlight a fundamental risk in the LLM-as-a-judge paradigm. The inability to reliably follow provided references when they conflict with internal knowledge suggests that current evaluation protocols may be flawed when applied to tasks involving evolving facts or counterfactual scenarios. The paper motivates the development of new protocols and training methods that can enforce stronger adherence to provided references, rather than relying on the model's internal priors.

Note: The paper explicitly states that it does not propose a concrete mitigation strategy, focusing instead on diagnosing the failure mode. It also limits its scope to QA evaluation, noting that findings may not directly generalize to other reference-conditioned tasks like summarization or dialogue evaluation without further study.