💬 NLP

Evaluating Communicative Belief Updates in Large Language Models via Implicature Recognition and Cancellation

This paper introduces the first expert-annotated dataset for implicature cancellation to evaluate large language models' ability to recognize and update unspoken beliefs, revealing that current LLMs lag behind human performance in these pragmatic reasoning tasks.

Original authors: Cesare Spinoso-Di Piano, Verna Dankers, Marius Mosbach, Jackie Chi Kit Cheung

Published 2026-07-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Cesare Spinoso-Di Piano, Verna Dankers, Marius Mosbach, Jackie Chi Kit Cheung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're playing a game of charades, but instead of acting out words, you're having a conversation where the most important parts are the things you don't say. In the world of human language, we are constantly negotiating a shared reality, a "common ground" where we both agree on what is true. We do this not just by stating facts, but by dropping hints, or implicatures. If I say, "Some of the cookies are gone," I'm hinting (without saying it directly) that not all of them are gone. You understand this instantly because your brain is a master of reading between the lines. But sometimes, I might change my mind and say, "Actually, I think they're all gone." That's called cancellation, where I take back the hint I just gave. This ability to instantly update our shared beliefs based on subtle, unspoken cues is the glue that holds human conversation together.

Now, imagine teaching a super-smart robot to play this game. We've built massive AI models called Large Language Models (LLMs) that can write poetry, solve math problems, and chat like humans. But do they truly understand the invisible dance of hints and retractions? Do they get it when a conversation shifts, or are they just guessing based on what they've seen before? This is the big question researchers are asking. If AI can't handle these tiny, unspoken updates, it will struggle to have a real, fluid conversation with us, often missing the point or getting confused when we change our minds.

In this paper, the researchers decided to put these AI models to the test. They created a brand-new playground called IMPLICATUREX, a dataset of 271 carefully crafted conversation snippets. Think of it as a series of mini-movies where a character drops a hint, and then immediately takes it back. They asked both human volunteers and various AI models to watch these clips and answer: "Did the hint happen? Did the retraction work? Did the character's belief change?"

The results were a bit of a reality check for the AI. While the smartest AI models were surprisingly good at spotting simple, textbook-style hints (like the "some vs. all" cookie example), they stumbled badly when the conversation got messy and real. When the hints came from natural, everyday chats—like two friends talking about a party or a hobby—the AI models often failed to understand the hint in the first place, or they completely missed the retraction. In fact, in the most natural scenarios, the AI's performance was barely better than random guessing.

The researchers also ran some sneaky control experiments to figure out why the AI was struggling. They found that when the AI got a hint right, it wasn't always because it understood the conversation; sometimes, it was just relying on a "prior belief," like a student who memorized the answer key without reading the question. For instance, if the AI saw the word "some," it might automatically assume "not all" without even listening to the rest of the sentence. Furthermore, when the AI was asked to update its beliefs after a retraction, it was much better at sticking to its original guess than at changing its mind. It's as if the AI is stubborn, preferring to ignore the new information rather than do the mental gymnastics required to update its understanding.

Ultimately, the study suggests that while our AI models are getting better at mimicking human speech, they haven't quite mastered the art of human thinking in conversation. They can follow the script, but they still lag behind us when it comes to the unspoken, shifting beliefs that make real communication work. The paper doesn't claim the AI is broken, but it does suggest that until these models can handle the nuance of cancelling a hint in a natural chat, they aren't quite ready to be true conversational partners.

Technical Summary: Evaluating Communicative Belief Updates in Large Language Models via Implicature Recognition and Cancellation

Problem Statement

Human communication relies heavily on the negotiation of unspoken beliefs and their dynamic updates within a "common ground." These updates often occur implicitly through implicatures (implied meanings) and implicature cancellations (the weakening or negation of those implied meanings). While Large Language Models (LLMs) have demonstrated proficiency in world knowledge and logical reasoning, their ability to manage these subtle, pragmatic belief updates in a communicative context remains underexplored. Specifically, it is unclear whether LLMs can recognize unspoken beliefs triggered by implicatures and subsequently update or cancel those beliefs when presented with contradictory information, a fundamental requirement for successful human-AI interaction.

Methodology

Dataset Construction: IMPLICATUREX

To address the lack of evaluation data, the authors constructed IMPLICATUREX, the first expert-annotated dataset of implicature cancellations. The dataset comprises 271 items categorized into three types:

Scalar Implicatures (46 items): Derived from the Switchboard Corpus, focusing on the pragmatic relationship between terms like "some" and "all" (e.g., "Some places nail them for tax" implying "Not all...").
Discourse Implicatures (31 items): Extracted from Wall Street Journal articles using implicit causal discourse relations from the Penn Discourse Tree Bank (PDTB).
Conversational Implicatures (194 items):
- Synthetic (144 items): Two-turn dialogues constructed from existing datasets where a triggering utterance introduces a belief and a cancelling utterance negates it.
- Naturally-Occurring (50 items): Real-world exchanges from the Switchboard Corpus where cancellations occur naturally within the dialogue.

Annotation Process:

Expert Annotation: Two professionally trained linguists annotated the items for the plausibility of the implicature and the correctness of the cancellation. Items with low agreement or implausible interpretations were replaced or discarded, resulting in the final set of 271 items.
Crowdsourcing: 90 participants provided likelihood judgments on a 7-point Likert scale for each item (with and without the cancelling utterance) to establish a human baseline for belief update accuracy.

Experimental Setup

The study evaluated LLMs on three specific tasks using a multiple-choice prompt template to estimate the probability of a belief $b$ given a context $c$ and utterance sequence $u$ :

Implicature Recognition: Determining if the model favors the implicated belief over its negation after the triggering utterance.
Cancellation Recognition: Determining if the model decreases the probability of the belief after observing the cancelling utterance.
Belief Update: A joint task requiring the model to first recognize the implicature and then successfully update (cancel) it.

Models Evaluated:
The study tested a range of open-weight models (Gemma 3, Llama 3.1/3.2/3.3, Qwen 2.5/3) and closed-source models (GPT-5.2/5.4), including variants with and without "reasoning" capabilities.

Control Experiments:

Prior Common Ground: Measured model performance without the triggering utterance to isolate reliance on prior biases.
Form Control (IMPLICATURE⊥): Replaced natural cancellations with explicit negations (e.g., "I am not saying X") to test if failures stem from pragmatic reasoning or negation interpretation.
Update Type Control: Tested model responses to strengthening utterances ( $u^+$ ) and unchanging utterances ( $u^\approx$ ) to see if models struggle specifically with changing beliefs versus maintaining them.

Key Results

1. Implicature Recognition

Scalar and Discourse Implicatures: Top-tier models (e.g., Gemma 3 27B, Llama 3.3 70B) matched or slightly exceeded human performance (accuracy ~0.90–1.0).
Conversational Implicatures: Models struggled significantly, particularly with naturally-occurring examples. The best model (GPT-5.4 Thinking) achieved 0.81 on synthetic examples but only 0.60 on naturally-occurring ones. Many models performed worse than random chance on naturally-occurring conversational implicatures.
Prior Bias: Control experiments revealed that models often possess strong priors for certain implicatures (e.g., "some" implies "not all") even without seeing the context, suggesting high recognition scores may reflect memorized patterns rather than genuine pragmatic reasoning.

2. Cancellation Recognition

Models generally performed closer to human levels on cancellation recognition for scalar and discourse implicatures.
However, performance dropped significantly for naturally-occurring conversational cancellations. The best model (Qwen 3 32B Thinking) achieved 0.84 accuracy compared to human 0.92.
Explicit Negation: Even when cancellations were made explicit (e.g., "I am not saying X"), models failed to reach perfect accuracy, indicating that difficulties extend beyond simple negation interpretation to broader pragmatic competence.

3. Belief Updates

Joint Task Difficulty: When requiring both recognition and cancellation, model performance fell sharply. For naturally-occurring conversational implicatures, the best model achieved only 0.40 belief update accuracy, compared to 0.72 for humans.
Maintenance vs. Change: Models demonstrated a strong bias toward maintaining existing beliefs. They performed well when presented with irrelevant follow-up utterances (unchanging) but struggled significantly when required to update (cancel) or strengthen beliefs. For instance, GPT-5.4 maintained beliefs with 0.96 accuracy on naturally-occurring examples but dropped to 0.32 when asked to update them.

Significance and Claims

The paper claims that current LLMs have not yet reached human-level understanding of unspoken beliefs and the dynamic belief updates inherent in human communication.

Gap in Pragmatic Reasoning: While models excel at static knowledge retrieval and logical deduction, they lag in the "negotiation" of meaning. They struggle to update beliefs in realistic, naturally-occurring scenarios, often relying on prior biases rather than contextual evidence.
Limitations of Current Architectures: The inability to consistently cancel implicatures, even when explicitly negated, suggests that LLMs may lack a robust mechanism for revising the common ground. They appear better at maintaining a state of belief than at revising it.
Dataset Contribution: The introduction of IMPLICATUREX provides a necessary benchmark for evaluating the "communicative competence" of LLMs beyond standard NLP tasks, highlighting a critical area where current models fall short of human interaction standards.

The authors conclude that while LLMs show promise in recognizing simple pragmatic inferences, their capacity to manage the fluid, unspoken negotiations of belief required for successful human-AI dialogue remains limited.