Do Language Models Know Theo Has a Wife? Investigating the Proviso Problem

This paper introduces a diagnostic dataset and evaluation framework to investigate how language models handle the proviso problem in pragmatics, revealing that while models align with human judgments, they rely on shallow pattern matching rather than genuine semantic or pragmatic reasoning.

The Gist

Imagine you are teaching a robot how to understand human conversation. You want to see if the robot truly "gets" the hidden meanings we take for granted, or if it's just memorizing patterns like a parrot.

This paper is about a specific puzzle in language called the "Proviso Problem."

The Puzzle: The "Theo" Riddle

Let's look at a simple sentence:

"If Theo hates sonnets, so does his wife."

What does this sentence actually assume to be true?

• The Robot (Formal Logic) says: "I can only be sure Theo has a wife if he actually hates sonnets. If he doesn't hate sonnets, maybe he's a bachelor. So, the fact that he has a wife is conditional."
• The Human says: "Wait, the sentence implies Theo definitely has a wife, no matter what. The 'if' part only applies to the hating of sonnets, not the existence of the wife."

Humans naturally fill in the missing piece (Theo has a wife) without thinking. This is called presupposition. The "Proviso Problem" is the gap between what formal logic says should happen and what humans actually do.

The Experiment: The "Magic Mirror" Dataset

The researchers built a giant "magic mirror" (a dataset of 8,500 sentences) to see how Language Models (like RoBERTa, LLaMA, and Gemma) handle this riddle.

They created four types of tests, like different levels of a video game:

1. The Baseline: Standard sentences (e.g., "If Randolf is a carpenter, he uses his tools").
2. The Twist (Structure): Changing the sentence shape (e.g., "If A and B, then C" or "Either A or B").
3. The Swap (Meaning): Swapping words for similar-sounding but different-meaning words (e.g., changing "wetsuit" to "garment").
4. The Distraction (Context): Changing the story so the two parts of the sentence don't make sense together logically.

The Results: Parrots vs. Philosophers

The researchers didn't just ask the models "What's the answer?" They also used X-ray vision (a technique called explainability) to see which words the models were looking at when they made their decision.

Here is what they found:

1. The Models are "Human" on the Surface, but "Robotic" Inside
When asked the simple riddles, the models got the right answer almost 100% of the time. They agreed with humans that "Theo has a wife."

• The Catch: When the researchers used X-ray vision, they saw the models weren't thinking about the meaning of "wife" or "Theo." They were just looking at the position of the words. It's like a student who passes a math test by memorizing the shape of the numbers rather than understanding addition.

2. The "Magic Word" Trap
In one test, the researchers swapped a key word.

• Original: "If Matt is a scuba diver, he'll bring his wetsuit." (Implies: Matt has a wetsuit).
• Swapped: "If Matt is a scuba diver, he'll bring his garment." (Implies: Matt has a garment).
• The Twist: The hypothesis was still "Matt has a wetsuit."

Logically, the sentence no longer proves he has a wetsuit. The answer should change from "Yes" to "Maybe/No."

• The Result: The models mostly failed. They kept saying "Yes, he has a wetsuit" because they saw the word "scuba diver" and the word "wetsuit" in the hypothesis, ignoring that the sentence actually said "garment." They were matching patterns, not reading the story.

3. The "Over-Student" Effect
When the models were trained on a specific set of examples, they got too good at memorizing the training data.

• They learned a weird rule: "If the story parts are related AND the word 'again' is used, the answer is 'No'."
• When the researchers changed the story slightly to break that rule, the models got confused and failed, even though the logic was simple. They were so focused on the training pattern that they couldn't adapt to a new situation.

The Big Takeaway

Think of these Language Models like brilliant actors who have memorized the script but haven't read the play.

• They can recite the lines perfectly and sound very human.
• They can predict what comes next based on what they've heard before.
• But, they don't truly understand the logic or the context behind the words. If you change a single word that breaks the pattern, they often stumble because they are relying on "shallow heuristics" (surface-level tricks) rather than deep reasoning.

Why This Matters

This paper is a wake-up call. Just because a model gets a high score on a test doesn't mean it understands language the way humans do. To build truly smart AI, we need to stop just checking the final answer and start looking at how the model thinks. We need to teach them to understand the "Theo" riddle, not just memorize the answer key.

Technical Summary

Here is a detailed technical summary of the paper "Do Language Models Know Theo Has a Wife? Investigating the Proviso Problem."

1. Problem Definition: The Proviso Problem

The paper addresses a fundamental issue in pragmatics and formal semantics known as the proviso problem.

• The Phenomenon: In conditional sentences containing presupposition triggers (e.g., "If Theo hates sonnets, so does his wife"), formal semantic theories predict that the presupposition ("Theo has a wife") projects conditionally. That is, the sentence implies: If Theo hates sonnets, then Theo has a wife.
• The Human Reality: In practice, human speakers typically accommodate the presupposition unconditionally. They interpret the sentence as asserting that Theo has a wife, regardless of whether he hates sonnets.
• The Gap: This discrepancy between theoretical predictions (conditional projection) and human pragmatic reasoning (unconditional accommodation) has remained unresolved. The authors investigate whether Large Language Models (LLMs) align with formal theory or human intuition when processing these structures.

2. Methodology

The authors reformulated the proviso problem as a Natural Language Inference (NLI) task and introduced a diagnostic framework combining classification accuracy with explainability techniques.

A. Dataset Construction (PROVISER)

The authors created a new dataset of approximately 8,500 sentence pairs, expanding upon the existing CONFER dataset. The dataset is structured into four subsets to test specific variables:

1. Subset 1 (Original): 900 baseline examples from CONFER.
2. Subset 2 (Structural Variation): Modifies sentence structure (Conjunction, Disjunction, Belief Embedding) to test if presupposition projection holds across different syntactic embeddings.
3. Subset 3 (Trigger–Hypothesis Relatedness): Manipulates the semantic relationship between the trigger and the hypothesis (Related, Somewhat Related, Unrelated) to test if models rely on semantic content or surface position.
4. Subset 4 (Context–Trigger Relatedness): Alters the logical coherence between the antecedent and consequent (Related vs. Unrelated contexts) while keeping the trigger constant.

Labeling Scheme:

• Human Labels: Based on empirical human judgments (typically Entailment/E for unconditional presuppositions).
• Theory-Based Labels: Based on formal semantic theory (typically Neutral/N, as the presupposition is conditional, not absolute).

B. Models Evaluated

Four models were tested:

• RoBERTa-large-MNLI and DeBERTa-large-MNLI (NLI-specific, fine-tuned on MultiNLI).
• Llama-3.2-1B and Gemma-3-1B (General-purpose open-weight models).

C. Evaluation Metrics

Beyond standard classification accuracy, the study employed explainability analyses to determine how models reach decisions:

• Integrated Gradients (IG): Measures token-level attribution to see if models focus on the presupposition trigger (e.g., "his wife") or superficial patterns.
• Trigger IG Ratio: Quantifies the proportional influence of trigger words relative to all tokens.
• Attention Analysis: Specifically for Subset 4, measured attention flow between the trigger noun phrase ($K_1$) and the contextual phrase ($K_2$) to assess pragmatic linking.

3. Key Contributions

1. Reformulation: Transformed the abstract proviso problem into a computationally testable NLI task.
2. Diagnostic Dataset: Created the first large-scale dataset (8.5k examples) specifically designed to probe presupposition projection with controlled structural, semantic, and contextual variations.
3. Multi-Method Evaluation: Pioneered the use of attribution-based explainability (IG and attention) alongside accuracy to distinguish between genuine pragmatic reasoning and shallow pattern matching in LLMs.

4. Key Results

• Alignment with Humans, Not Theory: In zero-shot and fine-tuned evaluations, models (especially RoBERTa and DeBERTa) achieved near-perfect accuracy against human labels (predicting unconditional presuppositions) but 0% accuracy against theory-based labels.
• Reliance on Shallow Heuristics:
  • Structural Robustness: Models maintained high accuracy across structural variations (Subset 2), suggesting they handle complex embeddings well.
  • Semantic Fragility: In Subset 3, when trigger phrases were semantically altered (e.g., changing "his wife" to "his friend" in a context where the hypothesis still claimed "Theo has a wife"), model accuracy dropped drastically (to 24–52%). Crucially, Integrated Gradients showed that models continued to assign high importance to the trigger's position even when the semantic content was broken. This indicates reliance on positional heuristics rather than semantic understanding.
• Overfitting to Training Data: In Subset 4, fine-tuned models failed on "Unrelated Context" examples involving the trigger "again." They incorrectly predicted Neutral (N) instead of Entailment (E). Attention analysis revealed that models had learned a spurious correlation from the training set: "Related Antecedent + 'again' = Neutral." When the context became unrelated, this learned pattern caused misclassification.
• Model Differences:
  • RoBERTa: Showed the highest Trigger IG ratios, indicating a strong focus on trigger words, which made it more robust to semantic changes than others (though still imperfect).
  • DeBERTa: Achieved high accuracy but with very low Trigger IG ratios (<1), suggesting it relied on other features or patterns, making it highly fragile when those patterns were disrupted.

5. Significance and Conclusion

The paper concludes that while LLMs appear to possess pragmatic competence (matching human judgments), this performance is superficial.

• Pattern Matching vs. Reasoning: Models solve the proviso problem by matching surface-level structural patterns (e.g., the presence of a possessive pronoun in a conditional) rather than engaging in genuine pragmatic reasoning or semantic accommodation.
• Limitations of Accuracy: High classification accuracy is misleading. A model can be 99% accurate on standard tests but fail completely when semantic relationships are perturbed, revealing a lack of true understanding.
• Future Directions: The authors argue that evaluating pragmatic reasoning requires diagnostic datasets paired with explainability tools. Future work should extend this to other pragmatic phenomena (e.g., implicature) and compare model behavior directly with psycholinguistic human processing data.

In summary, the study demonstrates that current LLMs "know" Theo has a wife not because they understand the pragmatics of the sentence, but because they have memorized the statistical association between specific trigger positions and unconditional entailment labels.