Evaluating Pragmatic Reasoning in Large Language Models: Evidence from Scalar Diversity

Technical Summary: Evaluating Pragmatic Reasoning in Large Language Models: Evidence from Scalar Diversity

Problem Statement
Evaluating pragmatic reasoning in Large Language Models (LLMs) remains a significant challenge due to the variability of model behavior across different evaluation methods. Previous research suggests a divergence between prompt-based judgments and a model's internal probability distributions, raising the question of whether observed pragmatic competence reflects underlying linguistic knowledge or is merely an artifact of task-induced behavior. While scalar implicature (the inference that a weaker term, e.g., "some," implies the negation of a stronger alternative, e.g., "not all") is a standard diagnostic for pragmatic ability, prior studies often rely on single scalar pairs or single evaluation paradigms. This study addresses the need to systematically compare evaluation methods to determine if LLMs possess stable pragmatic competence or if their performance is contingent on specific prompting and task structures.

Methodology
The study adopts the competence–performance framework (Chomsky, 1965), distinguishing between internalized knowledge (competence) and language use under specific conditions (performance). It utilizes scalar diversity—the phenomenon where the strength of scalar implicatures varies across different lexical scales (e.g., <some, all> vs. <warm, hot>)—as a graded diagnostic tool.

• Models: The study evaluates two distinct model families to compare architectural effects:
  • Flan-T5: An instruction-tuned encoder–decoder family (Small, Base, Large).
  • Qwen2: A decoder-only instruction-tuned family (0.5B, 1.5B, 7B parameters).
• Materials: The dataset was adapted from Ronai and Xiang (2024) and expanded using GPT-4o to include 60 lexical scalar pairs, resulting in 6,000 distinct items. Each item consists of an anchor sentence and two candidate interpretations: a "Yes-target" (pragmatic inference) and a "No-target" (logical interpretation).
• Evaluation Paradigms: Two primary methods were compared:
  1. Direct Probability Measurement: Calculating conditional probabilities $P(\text{candidate} | \text{anchor})$ to determine the model's internal preference without explicit instruction.
  2. Metalinguistic Prompting: Three prompt variations (MetaSimple, MetaInstruct, MetaComplex) requiring the model to explicitly judge or select the valid inference.
• Experimental Conditions:
  • Experiment A (Sentence Judgment): The model evaluates a single candidate sentence against the anchor (Yes/No).
  • Experiment B (Sentence Comparison): The model selects between two candidate sentences simultaneously.

Key Results
The results indicate that no single evaluation method consistently outperforms the other, and pragmatic behavior is highly sensitive to model architecture, size, and task structure.

• Divergence of Methods: Direct probability measurements and metalinguistic prompting produced only weakly correlated results ($r \approx .10 - .25$). In many cases, correlations were near zero or negative, suggesting the two methods capture fundamentally different aspects of model behavior.
• Accuracy Disparities:
  • Qwen Models: Exhibited a polarized pattern. Direct probability estimates were consistently very low, while metalinguistic prompting yielded extremely high accuracy (often near ceiling in Experiment B).
  • Flan-T5 Models: Showed more moderate patterns, with direct probability estimates remaining relatively higher and occasionally outperforming specific metalinguistic prompts.
• Scalar Diversity Gradients: The study analyzed item-level accuracy to detect gradients across the 60 scalar pairs.
  • Flan-T5: Generally displayed clearer gradients (slope values between -1 and -2) under metalinguistic prompting, particularly in larger models, indicating sensitivity to the varying strength of implicatures.
  • Qwen: Showed inconsistent patterns. While some conditions in Experiment A showed strong gradients, many conditions in Experiment B produced nearly flat curves (slope $\approx 0$), suggesting the model assigned uniform interpretations regardless of scalar strength.
• Task Structure: Experiment B (comparison-based) consistently yielded higher accuracy than Experiment A (single-judgment) across nearly all models, suggesting that comparative tasks provide stronger cues for pragmatic enrichment.
• Scaling Effects: Scaling effects were not uniform. Flan-T5 showed a positive trend where larger models performed better. Qwen models did not show a consistent scaling trend; larger models did not systematically outperform smaller ones and sometimes performed worse.

Key Contributions

1. Systematic Comparison: The study provides a direct comparison of direct probability and metalinguistic prompting within the same models for pragmatic reasoning, extending the competence–performance framework beyond local linguistic phenomena.
2. Scalar Diversity as a Diagnostic: By utilizing a broad range of scalar pairs rather than a single pair (e.g., <some, all>), the study offers a more fine-grained assessment of pragmatic competence, revealing that models do not uniformly capture the graded nature of scalar implicatures.
3. Architectural and Task Sensitivity: The findings highlight that pragmatic reasoning in LLMs is not a stable, monolithic capability but an interaction between internal probabilistic representations and external task framing.

Significance and Claims
The paper concludes that pragmatic reasoning in LLMs cannot be attributed to a single evaluation paradigm. Neither direct probability measurement nor metalinguistic prompting consistently captures a model's "true" pragmatic competence. Instead, observed pragmatic behavior reflects an interaction between internal probabilistic representations and prompt-induced responses.

The authors argue that the presence or absence of scalar diversity gradients depends heavily on the specific combination of model architecture, prompting strategy, and experimental design. Consequently, interpreting the pragmatic abilities of LLMs requires careful consideration of evaluation design, as a model may appear competent under one condition (e.g., specific prompts) while lacking the underlying sensitivity to scalar diversity in another (e.g., direct probability). This challenges the notion of a single, stable measure of pragmatic competence in LLMs.