The Generative AI Paradox on Evaluation: What It Can Solve, It May Not Evaluate

This paper reveals a significant performance disparity where Large Language Models excel at generation tasks but struggle with evaluation, often producing unfaithful judgments even in areas where they lack competence, thereby challenging the assumption that generative proficiency guarantees evaluative reliability.

The Gist

Imagine you hire a world-class chef to cook a gourmet meal. They are incredible at chopping vegetables, seasoning the sauce, and plating the dish. Everyone agrees they are a master of cooking.

Now, imagine you ask that same chef to act as a food critic for a blind taste test. You hand them a plate of food (which might be their own cooking or someone else's) and ask, "Is this delicious? Is it cooked correctly?"

You would expect the master chef to be the best judge possible, right? After all, they know exactly what good food tastes like.

This paper says: "Not so fast."

The researchers at KAIST and Georgia Tech discovered a strange phenomenon they call the "Generative AI Paradox." It turns out that just because an AI is amazing at creating answers (cooking), it doesn't mean it's good at checking answers (critiquing).

Here is the breakdown of their findings using simple analogies:

1. The "Chef Who Can't Taste" (The Paradox)

The researchers tested several AI models (like GPT-4 and others) on a trivia game.

• The Cooking Task (Generation): The AI had to answer questions like, "Where was actor Nigel Hawthorne born?"
• The Critic Task (Evaluation): The AI had to look at an answer (e.g., "Coventry") and decide if it was right or wrong.

The Shocking Result:

• Case A: The AI correctly cooked the answer ("Coventry"), but when asked to critique that same answer, it said, "No, that's wrong!"
• Case B: The AI cooked a wrong answer ("London"), but when asked to critique a different wrong answer, it said, "Yes, that's correct!"

It's like a chef who can perfectly bake a cake but then looks at a picture of that same cake and says, "That's definitely not a cake."

2. The "Overconfident Student" (Unfaithfulness)

The paper introduces the concept of Faithfulness. This asks: Does the AI judge based on what it actually knows?

Imagine a student taking a math test.

• If they get a problem wrong, they should admit, "I don't know the answer to this."
• But these AIs are like overconfident students who get a question wrong, but when they grade someone else's paper, they confidently mark the correct answer as "Wrong" because they are confused about their own knowledge.

The study found that these AIs often don't know what they don't know. Even when they are stumped and can't answer a question themselves, they rarely say "I don't know" when grading. Instead, they guess, and often guess confidently and incorrectly.

3. The "Inconsistent Judge" (Lack of Reliability)

The researchers also found that the AI judges are inconsistent.

• If you give the AI two very similar wrong answers, it might mark one as "Wrong" and the other as "I don't know."
• It's like a referee in a soccer game who blows the whistle for a foul in one game, but ignores the exact same foul in the next game.

This inconsistency means you can't fully trust the AI to be the final judge of truth, even if it is very good at writing stories or answering questions.

The Big Takeaway

For a long time, people assumed: "If an AI is smart enough to write a great essay, it must be smart enough to grade essays."

This paper proves that assumption is false.

• Generation is like performing a magic trick.
• Evaluation is like understanding the mechanics of the trick.

Just because an AI can perform the trick perfectly doesn't mean it understands the mechanics well enough to spot a fake trick.

Why does this matter?
As we start using AI to grade school papers, check medical diagnoses, or fact-check news, we need to be careful. We can't just assume the AI is a "super-judge" because it's a "super-writer." We need to build better systems that check the AI's grading, because the AI might be grading on a whim rather than on actual knowledge.

Technical Summary

Here is a detailed technical summary of the paper "The Generative AI Paradox in Evaluation: What It Can Solve, It May Not Evaluate."

1. Problem Statement

The paper challenges the prevailing assumption that Large Language Models (LLMs) proficient in generation tasks (creating text) are inherently equally skilled in evaluation tasks (assessing the quality of text).

• The Context: As LLMs like GPT-4 approach or exceed human performance in generation, there is a growing trend to use them as automated, reference-free evaluators to reduce the cost of human assessment.
• The Hypothesis: The authors posit that an LLM's ability to generate a correct answer does not guarantee its ability to correctly judge whether an answer (its own or another's) is correct.
• The Core Issue: This discrepancy is termed the "Generative AI Paradox," where models may exhibit superior generative skills but fail in basic evaluation, or conversely, accurately evaluate tasks they cannot solve themselves. This raises critical concerns regarding the faithfulness (whether scores reflect actual knowledge) and trustworthiness of LLMs as evaluators.

2. Methodology

The study utilizes a controlled experimental setup to compare generative and evaluative performance across multiple models.

• Dataset: The TriviaQA dataset (Joshi et al., 2017) was used, specifically a curated subset of 905 open-domain, free-form Question-Answering (QA) pairs with predefined "golden" answers. Ambiguous or time-sensitive questions were filtered out.
• Models Evaluated:
  • Evaluators/Generators: GPT-3.5, GPT-4, PaLM-2, and Vicuna-13b (open-source).
  • Role: Each model acted as both a "SOLVER" (generating answers) and an "EVALUATOR" (grading answers).
• Experimental Pipeline:
  1. Generation Phase (SOLVE): Models generated answers to the 905 questions using a zero-shot prompt. Human experts manually verified these answers to establish a ground truth (Correct/Incorrect).
  2. Rubric Generation: Models were asked to generate their own evaluation rubrics based on a provided scale (Correct, Partially Correct, Incorrect, I don't know).
  3. Evaluation Phase (EVALUATE): Models acted as judges, scoring the outputs of all models (including themselves) using their generated rubrics. This was a reference-free approach (no golden answers provided to the evaluator).
  4. Metric Calculation:
     • Evaluation Accuracy: The alignment between the model's evaluation label and the human-verified ground truth.
     • Faithfulness: Analyzed by checking if a model's evaluation of its own answers (or others') aligned with its own generative performance.

3. Key Contributions

• Empirical Evidence of the Paradox: The paper provides quantitative evidence that high generative capability does not linearly correlate with high evaluative capability.
• Definition of "Unfaithful Evaluation": The authors identify specific failure modes where models evaluate answers in areas where they lack competence, or fail to recognize their own errors.
• Reference-Free Evaluation Analysis: Unlike previous studies that provided golden answers to evaluators, this study tests the robustness of LLMs in a realistic, reference-free setting, revealing significant reliability gaps.
• Faithfulness Metrics: The study introduces a framework to measure whether a model's scoring is based on its actual internal knowledge or is inconsistent/sycophantic.

4. Key Results

A. Performance Disparity

• Generation vs. Evaluation: All models showed a drop in performance when switching from generation to evaluation.
  • GPT-4: Generation accuracy ~88%, Evaluation accuracy ~80%.
  • GPT-3.5: Generation accuracy ~79%, Evaluation accuracy ~62%.
  • PaLM-2: Generation accuracy ~66%, Evaluation accuracy ~70% (an anomaly where evaluation was slightly higher, likely due to specific error patterns).
  • Vicuna-13b: Generation accuracy ~46%, Evaluation accuracy ~52%.
• Conclusion: Even the most advanced models (GPT-4) failed to achieve 100% evaluation accuracy on questions they successfully solved, indicating a lack of self-awareness.

B. The Two Faces of the Paradox

1. Case 1 (Generation Correct, Evaluation Incorrect): The model generates the correct answer but incorrectly judges a correct answer (from itself or others) as "Incorrect." This suggests the model fails to recognize its own competence.
2. Case 2 (Generation Incorrect, Evaluation Correct): The model generates a wrong answer but correctly identifies a different model's correct answer as "Correct." This indicates the model can evaluate knowledge it does not possess, raising questions about the source of its evaluation logic.

C. Faithfulness and Consistency Issues

• Inconsistency in Self-Grading: Models often graded their own "Correct" generations as "Incorrect" or "I don't know" (e.g., GPT-4 marked ~7.7% of its own correct answers as non-correct).
• Sycophantic Bias: Evaluators frequently marked answers from other models as "Correct" even when those answers contradicted the evaluator's own (incorrect) generation, suggesting a bias to agree with others rather than rely on internal knowledge.
• Inability to Admit Ignorance: Despite having an "I don't know" option, models rarely selected it (<0.01% for most models). Even when models failed to solve a question during the generation phase, they rarely admitted ignorance during the evaluation phase, proceeding to grade answers anyway.
• Label Inconsistency: Similar types of incorrect answers (e.g., answers containing the correct entity but in a messy format) received inconsistent labels (e.g., "Incorrect" vs. "I don't know") from the same model.

5. Significance and Implications

• Reliability of Automated Evaluation: The findings suggest that using LLMs as "judges" for open-ended generation tasks is currently unreliable. The assumption that a model's generative excellence translates to evaluative expertise is flawed.
• Need for Scrutiny: The "Generative AI Paradox" highlights a critical gap in model alignment. Models may appear competent but lack the self-awareness to verify their own outputs or the consistency to apply their own rubrics.
• Future Directions: The paper calls for rigorous scrutiny of the faithfulness of model evaluations. Future research must explore whether this paradox holds across different tasks and develop methods to ensure evaluators are truly grounded in their knowledge rather than hallucinating scores.

In summary, the paper argues that LLMs are not yet trustworthy as autonomous evaluators because their ability to judge is not perfectly correlated with their ability to generate, and they frequently exhibit unfaithful behaviors regarding their own knowledge boundaries.