How Hypocritical Is Your LLM judge? Listener-Speaker Asymmetries in the Pragmatic Competence of Large Language Models

1. Problem Statement

Large Language Models (LLMs) are increasingly evaluated in two distinct roles:

• As Generators (Speakers): Producing contextually appropriate language.
• As Judges (Listeners): Evaluating the appropriateness or correctness of linguistic outputs (the "LLM-as-a-judge" paradigm).

Current evaluation practices often treat these roles as interchangeable proxies for general linguistic competence. The authors argue that this assumption is untested, particularly in pragmatics (the study of how context contributes to meaning). Drawing on psycholinguistic evidence that human language comprehension and production are related but non-identical tasks, the paper investigates whether LLMs exhibit a similar asymmetry. Specifically, does success in judging pragmatic adequacy correlate with the ability to generate pragmatically appropriate responses?

2. Methodology

The study employs a controlled experimental design comparing LLM performance across three specific pragmatic tasks. For each task, the authors constructed parallel "Speaker" and "Listener" setups using the same underlying items to enable direct, item-level comparisons.

Experimental Tasks

1. False Presuppositions:

   • Task: Handling questions containing false presuppositions (e.g., "Did voters find out that the AfD party is in favor of Ukraine...?" where the premise is false).
   • Speaker: Generate a response that rejects the false presupposition.
   • Listener: Judge whether a given response correctly rejected or incorrectly accepted the false presupposition.
   • Data: German datasets ("False Scenarios" and "False Claims").

2. Antipresuppositions:

   • Task: Adhering to Heim's "Maximize Presupposition!" (MP!) principle, where a stronger presupposition trigger (e.g., "the") is preferred over a weaker one (e.g., "a") in specific contexts.
   • Speaker: Select the correct article (definite vs. indefinite) to complete a sentence.
   • Listener: Judge whether a provided sentence continuation is appropriate given the context.
   • Data: German "fruit-story" paradigm.

3. Deductive Reasoning:

   • Task: Determining logical validity based on a set of premises.
   • Speaker: Generate the missing word in a conclusion to make it logically valid.
   • Listener: Judge whether a stated conclusion logically follows from the premises (True/False).
   • Data: English datasets adapted from cognitive psychology.

Models Evaluated

The study evaluated 14 contemporary multilingual LLMs, including:

• Open-weight models: LLaMA-3 (8B), Qwen-3 (8B, 14B), Phi-4 (14B), OLMo-2 (7B, 13B, 32B), Mistral families (7B, Mixtral-8x7B), and the specialized evaluator M-Prometheus-14B.
• Proprietary models: GPT-4o, GPT-4.1, GPT-5, and Claude Sonnet 4.5.

Metrics

• Aggregate Accuracy: Comparing overall performance in Speaker vs. Listener roles.
• Conditional Probability ($\Delta_{cond}$): Calculating $P(\text{correct generation} | \text{correct judgment}) - P(\text{correct generation} | \text{incorrect judgment})$. This measures whether a correct judgment on a specific item predicts a correct generation on that same item.

3. Key Contributions

• Systematic Asymmetry Detection: The paper provides the first systematic evidence that pragmatic evaluation (listening) and pragmatic generation (speaking) are weakly aligned in current LLMs.
• Item-Level Analysis: Unlike previous studies that only looked at aggregate scores, this work analyzes the relationship between judging and generating on a per-item basis, revealing that correct judgments do not reliably predict successful generation.
• Critique of LLM-as-a-Judge: The findings challenge the validity of using LLMs as automatic judges for pragmatic tasks without validating their generative capabilities, highlighting a "hypocrisy" where models can identify errors they cannot produce.

4. Key Results

A. Robust Listener-Speaker Asymmetry

Across all three tasks, a significant majority of models performed substantially better as listeners than as speakers.

• False Presuppositions: The asymmetry was most pronounced. Models like Mistral-7B and LLaMA-8B showed near-zero speaker accuracy but moderate listener accuracy (e.g., Mistral-7B improved from ~2% to ~30%). Even GPT-4o showed a gap on the more difficult "False Claims" dataset.
• Antipresuppositions: Many open-weight models (e.g., Qwen-3-14B, Mistral-7B) showed large listener advantages ($\Delta > 0.3$). Interestingly, some top-tier proprietary models (GPT-5, Claude-Sonnet-4.5) showed a reverse pattern (perfect speaker accuracy but lower listener accuracy), likely due to instruction-following nuances.
• Deductive Reasoning: Open-weight models (e.g., LLaMA-8B, OLMo-2) were significantly better at judging logical validity than generating valid conclusions.

B. Weak Predictive Power of Judgments

The conditional analysis ($\Delta_{cond}$) revealed that correct judgments do not predict successful generation:

• Negative Correlation: In Antipresuppositions, for most models, items that were judged correctly were less likely to be generated correctly ($\Delta_{cond}$ was often negative). This suggests that recognizing a violation does not help the model produce the correct alternative.
• Task Dependence: Positive coupling (where good judgment predicts good generation) was only observed in larger proprietary models for False Presuppositions and Deductive Reasoning, but was largely absent in open-weight systems and Antipresuppositions.

C. Instruction Following Issues

A significant portion of the "listener" failures in proprietary models (e.g., Claude-Sonnet-4.5) was attributed to instruction-following failures (e.g., failing to output the required single-word label), rather than a lack of pragmatic understanding. This highlights a reliability issue in using LLMs as judges.

5. Significance and Implications

• Evaluation Practices: The paper argues that benchmarks relying solely on "listener-style" tasks (classification, rating, or judging) are insufficient proxies for a model's true pragmatic competence. High scores in judging tasks should not be interpreted as evidence of robust generative abilities.
• LLM-as-a-Judge Reliability: The study cautions against the uncritical use of LLMs as evaluators. The observed asymmetry and instruction-following failures suggest that evaluation pipelines need to be validated per-task, as a model's ability to judge does not guarantee its ability to generate or even follow the formatting constraints required for judging.
• Theoretical Alignment: The findings mirror psycholinguistic theories of human language processing, suggesting that LLMs, like humans, possess dissociated mechanisms for comprehension and production.
• Future Directions: The authors call for integrated evaluation frameworks that assess generation and evaluation jointly, rather than treating them as interchangeable. They also suggest that future research should investigate the internal mechanisms causing these dissociations.

In conclusion, the paper demonstrates that current LLMs are often "hypocritical judges": they can frequently identify pragmatic errors in others' outputs but fail to avoid those same errors when generating their own language.