Unveiling the Limits of Large Language Models in Inferring Pragmatic Meaning from Non-Verbal Responses

Technical Summary: Unveiling the Limits of Large Language Models in Inferring Pragmatic Meaning from Non-Verbal Responses

Problem Statement
While Large Language Models (LLMs) have demonstrated significant progress in understanding verbal language and pragmatic phenomena such as implicature and indirect speech acts, existing research has largely overlooked the role of non-verbal behavior when it operates in isolation. In human communication, non-verbal cues (e.g., silence, facial expressions, gestures) often serve as the exclusive signals of communicative intent, conveying indirect meanings without verbal accompaniment. Current evaluations primarily assess LLMs on verbal inputs or treat non-verbal cues merely as contextual support for text. This study addresses the open question of whether LLMs can recognize and infer indirect intent solely from non-verbal responses, a capability critical for robust human-AI interaction where spoken language may be absent or limited.

Methodology
The authors present the first systematic evaluation of LLMs' ability to infer pragmatic meaning from dialogue responses composed exclusively of non-verbal elements. The study is structured around three research questions: (1) Can LLMs recognize indirect intent in non-verbal responses? (2) When and how do they fail? (3) How can this ability be improved?

• Data and Categories: The evaluation utilizes a custom dataset categorizing non-verbal responses into three types: Silence (deliberate lack of speech), Facial Expressions (represented via Unicode emojis), and Movements (descriptive actions in parentheses).
• Task Design: Models are presented with multiple-choice questions where they must select the option that best interprets the context and underlying intent of a conversation ending in a non-verbal response. Distractors are carefully designed to include misinterpretations, literal explanations, abstract descriptions, and lexically coherent but semantically divergent responses.
• Models Evaluated: Experiments cover 13 models across six families (Qwen, Mistral, Llama, Phi, and API-based models like GPT and Claude), ranging from 3B to over 100 billion parameters.
• Experimental Settings: The primary evaluation is conducted in a zero-shot setting. Additional analyses include Chain-of-Thought (CoT) prompting, few-shot in-context learning, and ablation studies removing explicit contextual instructions. A human baseline was established using undergraduate students from diverse national backgrounds.

Key Results

• Performance Gap: LLMs exhibit a substantial performance drop when processing non-verbal responses compared to verbal ones. While models often achieve near-human accuracy on verbal indirect speech acts, their accuracy on non-verbal tasks drops by up to 60 percentage points.
• Category Difficulty: Performance varies by category. "Movements" yielded the highest average accuracy (0.67), followed by "Facial Expressions" (0.51), with "Silence" performing the worst (0.45). The silence category proved particularly challenging, with models achieving only 0.45 accuracy compared to 0.91 for humans.
• Model Scale and Family: Larger parameter sizes generally correlate with improved performance (e.g., 70B models outperforming 3B models), but this trend is not universal across all families or API-based models. Even advanced models like GPT-4o and GPT-o3 struggle, performing comparably to open-source 14B models in these specific tasks.
• Error Analysis: The predominant source of error is "abstract explanations," where models provide vague descriptions that fail to address the core communicative intent. This is followed by "literal interpretations." The analysis suggests models tend to focus on surface-level features rather than inferring latent intentions.
• Behavioral Patterns: Layer-wise analysis using logit lenses reveals that the ability to interpret context is established in deeper layers. Correct predictions show high confidence convergence, while errors often manifest as confident misinterpretations or delayed, unstable convergence.
• Intervention Effects:
  • Chain-of-Thought (CoT): Results were mixed; while some models (e.g., Llama-70B) improved, others (e.g., GPT-4o) showed significant degradation, often defaulting to literal interpretations.
  • In-Context Learning: Few-shot prompting significantly improved performance, particularly for larger models and API-based models, suggesting that providing examples helps models align surface cues with underlying intentions.

Key Contributions

1. Systematic Evaluation: This work provides the first systematic evaluation of LLMs' ability to infer underlying intent from dialogue responses composed exclusively of non-verbal signals.
2. Empirical Evidence of Limitations: Comprehensive experiments across diverse model families demonstrate that LLMs face persistent challenges in inferring meaning from non-verbal cues, resulting in markedly diminished performance relative to their verbal counterparts.
3. Behavioral Insights and Mitigation: The study identifies specific error patterns (abstractness and literalism) and demonstrates that few-shot in-context learning can substantially enhance the alignment of surface-level cues with communicative intentions.

Significance
The paper argues that accurate understanding of non-verbal behavior is vital for advancing the pragmatic competence of LLMs, as these cues are universally recognized communicative signals. By revealing that current models struggle to interpret indirect intent without verbal scaffolding, the study highlights a critical gap in the current generation of LLMs. The findings suggest that while models possess strong verbal reasoning capabilities, they lack the necessary grounding to interpret non-verbal signals independently. The demonstration that in-context learning can mitigate these issues offers a pathway for improving model alignment with human communicative norms in multimodal or text-only scenarios where non-verbal cues are the primary mode of expression.

Limitations
The authors note that the evaluation is conducted exclusively in English and relies on pragmatic inferences assumed to be cross-cultural. Additionally, the study focuses on textual representations of non-verbal cues rather than raw multimodal inputs (e.g., video or audio), though the authors posit that the language model remains the final interpreter of intent regardless of the input modality.