Technical Summary: Pragmatic Theories Enhance Understanding of Implied Meanings in LLMs

Problem Statement

Human communication relies heavily on implied meanings—implicit intentions, unstated goals, and context-dependent interpretations that go beyond literal text. While Large Language Models (LLMs) have shown success in various language tasks, accurately interpreting these non-literal meanings remains a significant challenge. Existing approaches to enhance pragmatic reasoning in LLMs generally fall into two categories:

1. Post-training: Updating model parameters via supervised fine-tuning or preference optimization (e.g., DPO). This is costly, labor-intensive, and risks "catastrophic forgetting" of previously learned capabilities.
2. In-Context Learning (ICL): Using carefully designed prompts, such as Chain-of-Thought (CoT) or few-shot examples. However, many existing ICL methods rely on instance-specific hints or top-down guidance derived from correct answers, which limits their generalizability and requires ad hoc intervention for specific problem formats.

The authors identify a gap in instance-agnostic, zero-shot methods that can guide LLMs to interpret implied meanings without modifying parameters or providing specific problem-solving examples.

Methodology

The study proposes a prompting strategy that integrates summaries of established linguistic pragmatic theories into the model's prompt. The core hypothesis is that while LLMs possess latent knowledge of these theories within their parameters, they require a "bootstrap" to recall and apply this knowledge effectively for reasoning tasks.

The experiments utilize the PRAGMEGA dataset, which covers seven pragmatic phenomena. The study focuses on five non-literal meaning tasks: Deceits, Indirect Speech, Irony, Maxims, and Metaphor.

Prompting Strategies

The authors compare four primary prompting configurations in a zero-shot setting:

1. Baseline-1 (Simple): Explicit instruction to output only the final answer.
2. Baseline-2 (Chain-of-Thought/CoT): Instruction to "think step-by-step" without theoretical guidance.
3. Proposed-1 (Gricean Prompting): The prompt includes a brief overview of Gricean Pragmatics (Cooperative Principle and four maxims: Quantity, Quality, Relation, Manner). The model is instructed to reason according to this framework.
4. Proposed-2 (Relevance Theory Prompting): The prompt includes an overview of Relevance Theory (focusing on the balance between cognitive effects and processing effort). The model is instructed to reason according to this framework.

Additionally, the study tests "Short" variants of the proposed methods, where the theory is merely named (e.g., "Let's think in line with Gricean theory") without providing a detailed summary, to determine if the mere invocation of the theory is sufficient.

Experimental Setup

• Models: Experiments were conducted on both open-source models (LLaMa 3.1, Qwen 2.5 series) and proprietary closed models (GPT-4o, GPT-4o mini, phi-4).
• Control: To rule out confounding factors, the authors conducted additional experiments using:
  • Unrelated authoritative theories (X-bar theory, Computational Complexity, Graph Theory).
  • Fictional "pragmatic" theories (Majoritarian Consensus, Distance Primacy).
  • General planning templates (Plan-to-Solve).

Key Results

Performance Improvements

• Overall Accuracy: The proposed methods consistently outperformed both baselines across most models. The Gricean Prompting method achieved the highest gains, with an accuracy improvement of up to 9.6% (0.096) over the best baseline (Simple) on the phi-4 model.
• Human-Level Performance: On the GPT-4o model, the proposed methods enabled the model to surpass human performance scores reported in the original PRAGMEGA study. For phi-4, the method allowed the model to reach human-level accuracy where the baseline had previously lagged.
• Theory Comparison: Gricean Prompting generally outperformed Relevance Theory Prompting across most models. The authors speculate this may be due to Gricean concepts appearing more frequently in the models' training corpora, rather than an inherent superiority of the theory itself.
• Short Prompts: Even without detailed explanations, merely mentioning the theory names in the prompt (Short Prompting) yielded performance improvements of 1–3% over baselines in larger models, though the gains were smaller than with full summaries.

Phenomenon-Specific Analysis

• Irony: The proposed methods showed the most significant improvement for Irony. GPT-4o, which scored below humans on the baseline, reached human-level accuracy with Gricean prompting.
• Maxims: Gricean Prompting was particularly effective for tasks involving violations of Grice's maxims, as the theory directly addresses these mechanisms.
• Metaphor: Improvements were observed, particularly in models with 14B+ parameters. However, "Short" prompting was sometimes insufficient for metaphors, suggesting that detailed theoretical comprehension may be necessary for this specific phenomenon.
• Deceits: Effectiveness varied by model. The authors note that Gricean theory alone did not always capture the nuances of social deception, suggesting potential future work with Politeness Theory.

Error Analysis and Confounding Factors

• Error Patterns: The analysis revealed that the proposed methods were most effective in cases where the model needed to distinguish between literal and implied meaning (e.g., Maxims) or where standard CoT failed.
• Confounding Controls: The additional experiments confirmed that the performance gains were not due to:
  • Increased input/output length.
  • The mere presence of an authoritative-sounding theory (unrelated theories like X-bar theory often decreased performance).
  • The use of fictional theories (which yielded mixed or lower results compared to real pragmatic theories).
  • General planning templates (Plan-to-Solve).

Contributions and Significance

The paper claims the following contributions:

1. Instance-Agnostic Method: A simple, zero-shot prompting method that incorporates summaries of Gricean and Relevance theories to improve LLM performance on implied meaning tasks without requiring task-specific examples or parameter updates.
2. Theory Invocation Effect: Evidence that simply referencing the names of pragmatic theories in a CoT prompt can yield performance improvements, suggesting that LLMs possess latent pragmatic knowledge that can be activated by the right cues.
3. Targeted Efficacy: Demonstration that the method is particularly effective for interpreting Irony and utterances that flout Grice's maxims.

Limitations and Future Directions

The authors modestly acknowledge several limitations:

• Generalizability: The effectiveness of the method on broader, upstream tasks (e.g., dialogue systems, instruction following) remains unverified.
• Mechanism Clarity: While the method works, the precise "why" (e.g., whether it truly activates latent knowledge or merely shifts the probability distribution) requires further investigation.
• Language Scope: The study is limited to English. The authors note that pragmatic meaning varies across languages, and cross-lingual consistency needs verification.
• Dataset Complexity: The PRAGMEGA dataset, while useful, may lack the contextual richness of real-world pragmatic phenomena. The authors suggest future work should focus on datasets with more complex, extended contexts (e.g., based on sitcoms).

In conclusion, the paper posits that grounding LLM reasoning in established linguistic frameworks offers a cost-effective and generalizable path to enhancing the interpretation of implied meanings, bridging the gap between raw language generation and human-like pragmatic understanding.