💬 NLP

Presupposition and Reasoning in Conditionals: A Theory-Based Study of Humans and LLMs

This study reveals that while large language models can sometimes match human ratings on presupposition projection in conditionals, their performance often stems from surface pattern matching rather than genuine pragmatic reasoning, highlighting a disconnect between statistical alignment and true linguistic competence.

Original authors: Tara Azin, Yongan Yu, Raj Singh, Olessia Jouravlev

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tara Azin, Yongan Yu, Raj Singh, Olessia Jouravlev

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "If-Then" Puzzle

Imagine you are playing a game of logic with a friend. You say: "If John is a scuba diver, he will bring his wetsuit."

Your friend asks, "Does John actually have a wetsuit?"

The Human Answer: Most humans say, "Probably not yet. He only has a wetsuit if he is diving." The wetsuit is tied to the condition.
The Other Human Answer: Now imagine you say: "If John flies to London, his sister will pick him up."
The Human Answer: Here, humans say, "Yes, John definitely has a sister." The fact that he has a sister doesn't depend on the flight; it's just a fact about John.

This difference is called the "Proviso Problem." It's a tricky puzzle in linguistics about how we decide if a hidden fact (a "presupposition") is always true or only true if the "If" part happens.

The Experiment: Humans vs. AI

The researchers wanted to see if Large Language Models (LLMs)—the brains behind chatbots like the one you are talking to—can solve this puzzle the same way humans do.

They set up a "taste test" with two groups:

120 Humans: Real people who read sentences and rated how likely a hidden fact was to be true on a scale of 0 to 7.
4 AI Models: Different versions of AI (like GPT-5, Gemini, Llama, and Qwen) that did the exact same task.

They tested the AI in two ways:

The "Bare" Test: Just the sentence (e.g., "If John is a diver...").
The "Context" Test: The sentence plus a tiny bit of background info (e.g., "John is from Ottawa. If John is a diver...").

The Results: The "Imposter" AI

Here is what they found, broken down into simple metaphors:

1. Humans are Intuitive Detectives

Humans are great at this. They mix logic with common sense. If the "If" part makes the hidden fact more likely (like a diver needing a wetsuit), they lower their rating. If the "If" part has nothing to do with the fact (like flying to London and having a sister), they keep the rating high. They are sensitive to how relevant the two parts of the sentence are to each other.

2. The AI "Mimics" the Answer, But Not the Reasoning

This is the most surprising part.

The Small AI (Qwen): This model gave answers that looked very similar to the humans. If humans said "7," the AI said "6.8." It was a great mimic.
The Big AI (GPT-5, Gemini): These models gave answers that were less like humans. They were often too high or too low, missing the subtle shifts humans made.

The Twist: When the researchers asked the AIs to explain why they gave those answers (like asking a student to show their math work), a weird pattern emerged:

The Small AI (Qwen), which gave the best human-like answers, actually had terrible reasoning. It couldn't explain the logic properly. It was just guessing based on patterns it saw in its training data, like a parrot repeating a phrase it heard often without knowing what it means.
The Big AI (GPT-5), which gave worse human-like answers, actually had better reasoning. It could explain the rules of logic well, but when it came to the final number, it got it wrong compared to humans.

The "Checklist" Analogy

To figure this out, the researchers used a "Judge AI" (a referee) with a checklist. Imagine a teacher grading a student's essay.

The Checklist: "Did you identify the trigger? Did you check if the context matters? Did you explain the logic?"
The Result: The "Big AI" passed the checklist (it wrote a good essay). The "Small AI" failed the checklist (its essay was messy).
The Paradox: Even though the "Small AI" failed the logic test, its final score was closer to the human average.

The Conclusion: Pattern Matching vs. Understanding

The paper concludes that the AI models aren't actually "understanding" the language the way humans do.

Humans use a mix of logic, world knowledge, and relevance to decide if a fact is true.
AI seems to be doing surface-level pattern matching. It looks at the words "If John is a diver" and "wetsuit," sees that they often appear together in its training data, and guesses the number. It doesn't truly "know" that the wetsuit is conditional on the diving.

The Takeaway: Just because an AI gives you an answer that looks right (like a student who memorized the answer key), it doesn't mean it understands the lesson. In fact, the AI that gave the most "human-like" answers was actually the one relying the least on real logic.

What the Paper Does NOT Say

It does not say AI is useless.
It does not say AI will never understand language.
It does not suggest using this for medical diagnosis or legal advice.
It simply says: Right now, when it comes to this specific type of tricky logic puzzle, AI is faking it better than it thinks, and the "smartest" looking answers might be the most superficial.

Technical Summary: Presupposition and Reasoning in Conditionals: A Theory-Based Study of Humans and LLMs

Problem Statement
The paper addresses a critical gap in the evaluation of Large Language Models (LLMs) regarding their pragmatic competence, specifically in the domain of presupposition projection within conditional sentences. While theories of meaning and pragmatics have long debated the "proviso problem"—how presuppositions triggered in the consequent of a conditional (e.g., "If John is a scuba diver, he will bring his wetsuit") are interpreted as either unconditional or conditional on the antecedent—this phenomenon remains largely unevaluated in LLMs. Existing benchmarks often rely on simple entailment reasoning or classification tasks that fail to capture the complex interaction between semantics, discourse context, probabilistic reasoning, and world knowledge required to resolve these projections. The authors argue that current evaluations do not sufficiently distinguish between models that genuinely understand pragmatic constraints and those that rely on surface-level pattern matching.

Methodology
The study employs a parallel behavioral design comparing human judgments against four LLMs (GPT-5, Gemini-2.5-flash, Llama-3.1-8B-Instruct, and Qwen2.5-7B-Instruct) using a normed dataset of 90 conditional sentences.

Data Construction: The authors constructed a dataset based on 30 base propositions (e.g., having a wetsuit, having a sister) instantiated in three variants manipulating the logical and probabilistic relationship between the antecedent ( $A$ ) and the presupposition ( $p$ ). A norming study with 30 native English speakers established graded levels of $A$ - $p$ relevance (high, mid, low) and proposition probability.
Experimental Conditions: Both humans ( $N=120$ $N = 120$ ) and LLMs evaluated the likelihood of the presupposition being true on a 0–7 Likert scale under two conditions:
- Without-Context: Minimal framing.
- With-Context: Added identifying background information (e.g., "The person is from Toronto") to test how minimal context modulates evidential reliability and relevance reasoning.
LLM-as-a-Judge Framework: To move beyond accuracy metrics, the authors developed a theory-informed checklist grounded in formal semantics (e.g., Dynamic Semantics, Gricean Pragmatics, Accommodation Theory). A judge model (Claude-haiku-4) evaluated the Chain-of-Thought (CoT) reasoning traces of the target models against 52–59 binary criteria covering dimensions such as Accuracy, Context Integration, Pragmatic Criteria, Presupposition Handling, and Coherence.
Validation: A subset of the judge's evaluations was validated by human linguistics experts, achieving 89% inter-annotator agreement and 79.46% agreement with the judge model.

Key Results

Human Behavior: Human judgments integrate probabilistic and pragmatic cues non-additively. When the antecedent is irrelevant to the presupposition, humans rely heavily on prior probability (world knowledge). However, even partial relevance acts as a threshold that significantly boosts likelihood ratings, suggesting a sensitivity to relevance that overrides base rates in specific contexts. Contextual information refined this integration, leading to more graded distinctions between relevance levels.
Human-LLM Alignment: LLMs showed variable alignment with human patterns.
- Qwen2.5-7B-IT and Llama-3.1-8B-Instruct demonstrated the strongest correlation with human Likert ratings (Spearman $\rho \approx 0.30\text{--}0.38$ ) and the lowest Mean Absolute Error (MAE).
- GPT-5 and Gemini-2.5-flash showed weaker or inconsistent alignment, with GPT-5 failing to show significant correlation in either condition.
Reasoning vs. Performance (The Dissociation): A critical finding is the dissociation between behavioral alignment and reasoning quality.
- Models with the highest behavioral alignment to humans (e.g., Qwen2.5-7B-IT) often exhibited the lowest compliance with the theory-informed checklist. Their reasoning frequently relied on surface lexical associations rather than formal pragmatic constraints.
- Conversely, larger, proprietary models (GPT-5, Gemini) generated reasoning traces that were more consistent with theoretical expectations (higher checklist scores) but produced final likelihood judgments that deviated more significantly from human patterns.
Context Effects: Providing minimal context generally improved checklist compliance for smaller models but introduced "reasoning noise" for some, leading to reduced coherence in their explanations.

Key Contributions

Theory-Driven Benchmark: The paper introduces a controlled dataset and experimental design specifically targeting the proviso problem in conditionals, filling a gap in existing NLI and presupposition benchmarks which often lack pragmatic depth.
LLM-as-a-Judge for Pragmatics: It proposes and validates a checklist-based evaluation framework that decomposes pragmatic competence into interpretable, theory-grounded criteria, offering a more nuanced view of model reasoning than accuracy scores alone.
Empirical Evidence of Competence Gaps: The study provides empirical evidence that high behavioral alignment in LLMs may result from probabilistic pattern matching on training data rather than stable pragmatic competence, as evidenced by the disconnect between human-like ratings and theory-consistent reasoning.

Significance and Claims
The authors claim that their findings highlight the limitations of current LLMs in handling complex pragmatic phenomena. The paper argues that performance on tasks like presupposition projection may reflect "surface pattern matching" rather than genuine pragmatic understanding. By demonstrating that models which best mimic human judgments often lack coherent pragmatic reasoning, the study suggests that current benchmarks based solely on output accuracy are insufficient for evaluating semantic and pragmatic competence. The authors conclude that benchmarks grounded in linguistic theory are essential for distinguishing between superficial statistical alignment and true reasoning capabilities, and that future work must address the dissociation between behavioral similarity and theoretical validity.