💻 computer science

The Consistency Dilemma in LLMs: Generator-Evaluator Agreement and Vulnerability to Mistakes

This paper introduces a measure of generator-evaluator self-consistency to reveal a critical dilemma in large language models: while higher consistency in applying concepts is operationally useful, it paradoxically correlates with a greater vulnerability to mistakes in clinical settings, suggesting that consistent models are not necessarily safe for deployment.

Original authors: Marina Mancoridis, Zoë Hitzig

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Marina Mancoridis, Zoë Hitzig

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Self-Correction" Trap

Imagine you hire a very smart, highly educated assistant to do a job. You ask them to write a report, and then you ask them to read their own report and check for mistakes. You assume that if they are smart enough to write the report, they are smart enough to spot their own errors.

This paper investigates a specific type of AI (Large Language Models) that works exactly like this: it generates an answer, then immediately acts as a judge to evaluate that same answer. The researchers wanted to know: Does the AI use the same rules when it writes the answer as it does when it checks the answer?

They call this "Generator-Evaluator Self-Consistency."

The Experiment: The "Same-Actor" Test

To test this, the researchers didn't just ask the AI a question and see if it got the right answer. Instead, they set up a game where the AI had to play two roles at once:

The Generator: The AI is asked to create a new version of a question or a specific type of answer (e.g., "Make up a math problem where the answer is 'None of the above'").
The Evaluator: The AI is then asked to look at the thing it just made and decide if it followed the rules.

The Analogy: Imagine a chef who is asked to bake a cake with no sugar.

Role 1 (Generator): The chef bakes a cake and claims it has no sugar.
Role 2 (Evaluator): The chef tastes the cake and says, "Yes, this has no sugar."

If the chef is consistent, they will taste the cake and correctly say, "Wait, I accidentally put sugar in this."
If the chef is inconsistent, they might taste the cake, forget they put sugar in, and say, "Yes, this is sugar-free," even though it's not.

The researchers tested 10 different "frontier" AI models across nearly 500 different concepts (like medical rules, math principles, or financial logic) to see how often the AI's "Generator" brain and "Evaluator" brain agreed with each other.

The Surprise Finding: The Consistency Dilemma

The researchers expected that if an AI was very good at following rules consistently, it would also be safer and less likely to make mistakes. They thought: "If the AI is disciplined enough to use the same logic for writing and checking, it should be a reliable worker."

They were wrong.

They discovered a strange paradox, which they call the "Consistency Dilemma."

The Finding: The AI models that were most consistent (the ones that used the same logic for writing and checking) were actually more likely to make dangerous mistakes in real-world scenarios.
The Counter-Intuitive Result: The models that were less consistent (the ones that sometimes changed their minds or applied rules differently between writing and checking) were actually safer and made fewer validated errors.

The Metaphor:
Think of it like a security guard who is extremely rigid and follows a single rulebook perfectly.

The Rigid Guard (High Consistency): He sees a suspicious person, follows the rulebook, and lets them in because the rulebook didn't explicitly say "stop them." He is very consistent in applying the rule, but he makes a huge safety error.
The Flexible Guard (Low Consistency): He sees the same person, hesitates, checks the rulebook, then looks at the person's face, and decides to stop them. He wasn't perfectly consistent in his logic (he used the rulebook and his gut), but he prevented a mistake.

The paper found that in high-stakes areas like medicine, the "Rigid Guard" (the highly consistent AI) was more likely to miss critical safety warnings because it was too busy sticking to its own internal logic to notice the danger.

Why This Matters (According to the Paper)

The paper highlights a tension in how we use AI today:

We want consistency: We want AI agents that can write code, check their own code, and fix errors without getting confused. We want them to be stable.
But consistency creates blind spots: When an AI is too consistent in its own internal logic, it can become "stubborn." It might confidently apply a rule it generated itself, even if that rule is dangerous or wrong in a real-world context.

The researchers tested this using a dataset of real medical mistakes validated by doctors. They found that the AI models with the highest "self-consistency" scores were the ones that most frequently repeated these dangerous medical errors.

The Bottom Line

The paper concludes that being "consistent" does not automatically mean being "safe."

In fact, in the specific tests they ran, the models that were most consistent in applying their own concepts were the most vulnerable to making validated mistakes. This suggests that when we build AI systems that rely on the AI to check its own work, we need to be careful: a model that is too consistent might be confidently wrong, while a model that is slightly inconsistent might actually be safer because it's less rigid in its thinking.

Key Takeaway: Just because an AI agrees with itself doesn't mean it's right. Sometimes, the AI that changes its mind a little bit is the one that catches the error.

Technical Summary: The Consistency Dilemma in LLMs

Problem Statement

As Large Language Models (LLMs) are increasingly deployed in agentic pipelines, they are often tasked with generating outputs and subsequently evaluating, filtering, or revising those same outputs without external verification. This workflow relies on an implicit assumption: that the model applies relevant concepts consistently across its generation and evaluation roles. If a model employs one understanding of a concept during generation and a different one during evaluation, the review step becomes unreliable. The model might flag a sound output based on a criterion it ignored during generation, or accept a flawed output because it has ceased enforcing a constraint it previously applied.

While standard benchmarks measure a model's knowledge of concepts, they do not measure whether a model applies those concepts stably across different roles (generator vs. evaluator). Furthermore, it is unclear whether high self-consistency correlates with higher reliability in real-world deployment, particularly in high-stakes settings where ground-truth verification is costly.

Methodology

1. Generator-Evaluator Self-Consistency Framework

The authors propose a new measure, generator–evaluator self-consistency, to test the stability of concept application. The framework operates without external correctness labels by comparing a model's output against its own earlier use of a concept.

Structure: For a given concept $c$ , a model is first asked to generate an output meeting a specific criterion (Generator step). The same model is then asked to evaluate whether its generated output meets that criterion (Evaluator step).
Scoring: A deterministic scoring rule assigns a binary agreement label ( $Z=1$ if consistent, $0$ otherwise).
Metrics: The authors report a chance-corrected consistency rate ( $\hat{C}_{adj}$ ), normalizing raw agreement to account for the baseline probability of agreement given the test structure (e.g., random guessing in multiple-choice vs. binary judgments).

2. Taxonomy of Consistency Tests

The evaluation suite consists of 12 automated tests organized into three qualitative categories, applied across 491 concepts sourced from general-reasoning (MMLU, BBH) and deployment-focused (MedicalQA, GDPVal, FinanceQA) benchmarks:

MCQ Perturbations: Tests whether the model remains consistent when a multiple-choice question is minimally edited (e.g., changing the correct answer to "None of the above" or shifting the correct option to a different letter).
Rationale-Based Transformations: Tests whether the model can audit its own reasoning traces. This includes generating correct/incorrect answers with correct/incorrect/incomplete reasoning and then judging whether the reasoning supports the answer.
Ontological Checks: Tests whether the model maintains a coherent organization of concepts. This involves generating examples, non-examples, attributes, or properties and then validating them under related prompts.

3. Clinical Case Study: Mistake Vulnerability

To investigate the relationship between consistency and reliability, the authors utilize MedMistakes-Validated, a dataset of clinically realistic vignettes paired with physician-validated model errors.

Mistake Vulnerability ( $Y^*$ ): For each (model, concept) pair, the authors measure how often the model reproduces validated clinical failure modes. This is aggregated using a weighted scheme to avoid double-counting multi-tagged mistakes.
Statistical Analysis: A weighted fractional logit model is used to regress mistake vulnerability against self-consistency and benchmark performance (MedQA accuracy), controlling for baseline medical knowledge.

Key Results

1. Variation in Self-Consistency

There is substantial variation in self-consistency across 10 frontier models (including GPT-4o, GPT-5 variants, Claude Opus/Sonnet, Gemini, DeepSeek, and Mistral). This variation cannot be explained solely by performance on the source benchmark questions. For instance, models like Gemini 3 and GPT-4o achieved near-ceiling agreement on ontological tests, while others performed significantly worse on similar tasks.

2. The Consistency Dilemma

The central finding is a positive association between self-consistency and mistake vulnerability in a clinical setting.

Empirical Finding: Models that are more consistent in how they apply concepts across generation and evaluation are more likely to reproduce clinician-validated mistakes.
Statistical Significance: After controlling for benchmark performance, the coefficient for self-consistency is positive and significant ( $\hat{\beta}_2 = 1.116, p < 10^{-6}$ ).
Between-Model vs. Within-Model: This relationship holds between models (i.e., Model A is more consistent and more vulnerable than Model B) but does not predict vulnerability within a single model across different concepts. When model fixed effects are added, the association disappears, suggesting self-consistency is a model-level behavioral signature rather than a concept-level predictor of failure.

3. Specific Test Sensitivity

The association is driven significantly by the Correct for Incomplete Reason (CFIR) test. This test checks if a model generates a correct answer with under-specified reasoning and then correctly identifies that reasoning as insufficient. Models that fail to flag their own incomplete reasoning (inconsistent behavior) tend to be less vulnerable to validated mistakes, whereas models that are highly consistent (often correctly flagging incomplete reasoning) are paradoxically more prone to reproducing specific clinical failure modes.

Significance and Claims

The paper claims to identify a "consistency dilemma" in LLMs:

Operational Utility: Self-consistency is operationally desirable for agentic pipelines as it reduces drift and supports workflows like "LLM-as-a-judge" or self-refinement.
Reliability Trade-off: However, in the studied deployment setting (clinical), higher self-consistency is linked to greater vulnerability to known, validated mistakes.

Practical Implications:

Model Selection: Practitioners should not assume that a more self-consistent model is inherently safer. In high-stakes settings, highly consistent models may require more rigorous human oversight or external evaluation rather than relying on autonomous self-critique.
Diagnostic Tool: The proposed automated, black-box framework provides a scalable method to diagnose reliability risks driven by repeated, systematic mistakes, complementing traditional single-turn accuracy benchmarks.

Limitations:
The authors note that the deployment analysis is observational, not causal. The results are specific to a curated set of clinician-validated mistakes and may not generalize to all deployment errors. Additionally, the concept extraction relies on model-generated labels, which may introduce measurement error if questions involve multiple interacting concepts.

In conclusion, the paper argues that consistency is a distinct behavioral property from accuracy and that higher consistency does not automatically translate to higher safety in complex, real-world scenarios.