CEI: A Benchmark for Evaluating Pragmatic Reasoning in Language Models

This paper introduces the Contextual Emotional Inference (CEI) Benchmark, a dataset of 300 human-validated scenarios designed to evaluate large language models' ability to infer intended meaning beyond literal semantics by navigating ambiguous utterances across diverse power dynamics and pragmatic subtypes.

The Gist

Imagine you are at a dinner party. Someone says, "Oh, great job on the presentation," but they say it with a sigh, rolling their eyes, while the person who gave the presentation just made a huge mistake.

A computer program looking only at the words might think: "They are happy! They used the word 'great'!"
But a human knows the truth: They are actually furious.

This gap between what is said and what is meant is called pragmatic reasoning. It's the secret sauce of human communication. We use sarcasm, passive aggression, and polite lies all the time.

This paper introduces a new test called CEI (Contextual Emotional Inference) to see if Artificial Intelligence (AI) can figure out these hidden meanings. Here is the breakdown in simple terms:

1. The Problem: AI is Bad at "Reading the Room"

Current AI models are like brilliant students who have memorized every dictionary in the world but have never actually talked to a human being. They are great at literal facts but terrible at understanding:

• Sarcasm: Saying the opposite of what you mean.
• Passive Aggression: Being "nice" on the surface while being mean underneath.
• Strategic Politeness: Softening a blow to save face.
• Deflection: Changing the subject to avoid a hard truth.
• Mixed Signals: When words say one thing, but the situation says another.

2. The Solution: A "Social Detective" Test

The researchers created a dataset of 300 short stories (scenarios).

• The Setup: Each story gives you the context (e.g., "A boss and an employee are in a meeting"), the power dynamic (who is in charge?), and a specific sentence the speaker says.
• The Task: You have to guess: What is this person actually feeling?
• The Twist: The sentence might be "Sure, I'll handle the extra work this weekend." Is that sincere? Is it a polite "no"? Is it a passive-aggressive "I hate you"?

They hired 15 human students to act as "detectives" and label the emotions. They also gave the AI the same test.

3. The Results: Humans Struggle, and AI is Lost

Here is the surprising part: Even humans found this hard.

• When three different humans looked at the same scenario, they often disagreed.
• The "agreement score" was low (about 21%). This isn't because the humans were bad; it's because pragmatic meaning is fuzzy. Sometimes, "I'm fine" could mean "I'm sad," "I'm angry," or "I'm just tired," and there is no single right answer.
• The AI did even worse. The best AI model only got 25% of the answers right. Humans got about 54% right.
• The Analogy: If this were a driving test, humans are driving at 50 mph in the rain (a bit shaky, but getting there). The AI is driving at 10 mph, confused about which lane is which, and keeps hitting the curb.

4. Why is AI Failing?

The paper found that AI fails in specific ways:

• It takes things too literally. If you say "That's a great idea" sarcastically, the AI thinks you are happy.
• It ignores power dynamics. It doesn't understand that an employee saying "Sure" to a boss is different than a friend saying "Sure" to a friend.
• It can't handle the "fuzzy" stuff. When humans disagree on the emotion, the AI just guesses the most common negative emotion (like Anger or Sadness) and moves on. It doesn't have the intuition to say, "Hmm, this is ambiguous."

5. Why Does This Matter?

You might think, "So what? AI is just bad at jokes." But this skill is crucial for real-world safety and utility:

• Mental Health: If a chatbot is talking to someone depressed, and the person says "I'm fine" (but means "I'm not"), the AI needs to know to ask, "Are you sure?"
• Workplace Safety: If an HR tool scans emails for "toxic" behavior, it needs to spot passive-aggressive emails that look polite on the surface but are actually hostile.
• Accessibility: For people who struggle with social cues (like those with autism), AI tools that can translate "polite lies" into "real feelings" could be a huge help.

The Bottom Line

This paper is a reality check. It shows that while AI is getting smarter at math and writing, it is still socially clumsy. It can write a poem, but it can't tell if you are being sarcastic.

The researchers released this test to the public so other scientists can try to fix it. They are essentially saying: "Here is a mirror showing exactly where our AI is blind. Let's work on teaching it to read the room."

Technical Summary

Here is a detailed technical summary of the paper "CEI: A Benchmark for Evaluating Pragmatic Reasoning in Language Models."

1. Problem Statement

Large Language Models (LLMs) struggle with pragmatic reasoning: the ability to infer intended meaning that diverges from literal semantics. While models perform well on surface-level sentiment analysis, they fail to interpret indirect speech acts (e.g., sarcasm, passive aggression, strategic politeness) where context, social roles, and power dynamics are critical. Existing benchmarks either focus on narrow phenomena (e.g., sarcasm detection only), lack explicit social context (power relations), or rely on single gold-standard labels that ignore the inherent ambiguity of pragmatic inference.

2. Methodology

Dataset Design (CEI Benchmark)

The authors introduce the Contextual Emotional Inference (CEI) benchmark, consisting of 300 expert-authored scenarios.

• Structure: Each scenario includes a situational context (2–4 sentences), explicit speaker/listener roles with defined power relations, and an ambiguous utterance.
• Subtypes: The data covers five pragmatic subtypes:
  1. Sarcasm/Irony
  2. Mixed Signals
  3. Strategic Politeness
  4. Passive Aggression
  5. Deflection/Misdirection
• Contexts: Drawn from four settings: Workplace, Family, Social, and Service.
• Power Relations: Three configurations: Peer-to-Peer (72%), Higher-to-Lower (20%), and Lower-to-Higher (7%).
• Annotation Task: Annotators must infer the speaker's primary emotion using Plutchik's 8 basic emotions and provide Valence-Arousal-Dominance (VAD) ratings on 7-point scales.

Annotation Process & Quality Control

• Annotators: 15 undergraduate students (fluent English speakers) annotated the dataset. Each scenario was labeled by 3 independent annotators (900 total annotations).
• 4-Level Quality Control Pipeline:
  1. Schema Validation: Automated checks for JSON structure and valid enums.
  2. Statistical Consistency: Detection of "straight-lining" (assigning the same label >80% of the time), timing outliers (<3s or >600s), and self-contradictions (e.g., positive emotion with negative valence).
  3. Agreement Analysis: Calculation of Fleiss' $\kappa$ per subtype. Scenarios with total disagreement or high VAD divergence were flagged.
  4. Expert Adjudication: A meta-annotator reviewed flagged items (15.7% of data). In cases of three-way splits, the meta-annotator applied heuristics based on valence polarity and context misreading to establish a "gold standard," while preserving the full distribution of human judgments.

Evaluation Framework

• Models: 7 LLMs were evaluated (4 commercial APIs: GPT-5-mini, Claude Sonnet 4.5, Gemini 2.5 Flash; 3 open-weight: Llama-3.1-70B, DeepSeek-V3, Qwen2.5-7B).
• Prompting Modes: Zero-shot, Chain-of-Thought (CoT), and Few-shot (3-shot).
• Handling OOV Labels: Models frequently generated Out-of-Vocabulary (OOV) emotion labels. The authors implemented a taxonomy harmonization pipeline mapping these to Plutchik's wheel using the NRC Emotion Lexicon and wheel adjacency.
• Metrics: Accuracy (exact match) and Macro-F1, with breakdowns by subtype and power relation.

3. Key Contributions

1. CEI Benchmark: A dataset of 300 scenarios integrating pragmatic subtypes, explicit power dynamics, and multi-annotator labels with documented agreement levels. It is the first benchmark to jointly evaluate emotion inference across multiple forms of indirect speech with social context.
2. Robust Quality Control Pipeline: A novel 4-stage pipeline that distinguishes between genuine pragmatic ambiguity (low agreement is expected) and annotation errors, allowing for the release of full per-annotator labels rather than just a single gold standard.
3. Standardized Evaluation Protocol: Baselines across 7 models and 3 prompting modes, establishing that current models fail significantly on pragmatic reasoning tasks compared to humans.

4. Key Results

Human Performance

• Inter-Annotator Agreement: Low, as expected for this task. Overall Fleiss' $\kappa = 0.21$ (ranging from 0.06 for Deflection to 0.25 for Sarcasm).
• Agreement Distribution:
  • Unanimous agreement: 14.3%
  • Majority agreement: 54.3%
  • Three-way splits: 31.3%
• Insight: Even humans struggle to agree on the emotional intent of indirect speech, confirming the task's inherent difficulty. However, even in disagreement, annotators often agreed on valence polarity (e.g., all chose negative emotions), suggesting systematic rather than random disagreement.

Model Performance

• Accuracy: Current LLMs perform poorly, with a best accuracy of 25.0% (Llama-3.1-70B and Qwen2.5-7B), compared to 54.3% human majority agreement.
• Random Baseline: 12.5% (8-class classification).
• Prompting Impact: Neither Chain-of-Thought (CoT) nor Few-shot prompting meaningfully improved performance (Mean accuracy remained ~20%).
• Subtype Variance:
  • Hardest for Models: Sarcasm/Irony (15.2% avg accuracy) and Passive Aggression (16.0%).
  • Easiest for Models: Strategic Politeness (25.5%) and Mixed Signals (22.1%).
  • Dissociation: Interestingly, Sarcasm (easiest for humans, $\kappa=0.25$) is hardest for models, while Deflection (hardest for humans, $\kappa=0.06$) is middling for models. This suggests models and humans fail for different reasons.
• Error Patterns: Models tend to over-predict high-frequency negative emotions (Anger, Sadness) and struggle with rare emotions (Disgust, Anticipation).

5. Significance and Implications

• Gap in Pragmatic Competence: The ~29 percentage point gap between human majority agreement and the best model highlights a fundamental limitation in current LLMs: they cannot effectively integrate social context and power dynamics to resolve pragmatic ambiguity.
• Diagnostic Utility: The benchmark reveals specific failure modes. For instance, models fail at sarcasm despite its high human agreement, suggesting they lack the ability to detect tonal incongruity or shared expectations.
• Beyond Binary Labels: The release of per-annotator labels and VAD ratings supports soft-label training and calibration-aware evaluation, acknowledging that pragmatic meaning is often a distribution of valid interpretations rather than a single truth.
• Broader Impact:
  • Positive: Improving pragmatic reasoning is crucial for mental health chatbots (detecting distress masked by "I'm fine"), accessibility tools for neurodivergent individuals, and conflict mediation.
  • Negative Risks: These capabilities could be misused for workplace surveillance, political manipulation, or creating deceptive agents that exploit power dynamics. The authors advocate for open release and strict documentation of limitations to mitigate these risks.

In conclusion, CEI establishes that pragmatic reasoning remains a "hard" frontier for AI, distinct from standard sentiment analysis, and requires models to deeply understand social hierarchies and indirect communication strategies.