💬 NLP

Why are language models less surprised than humans? Testing the Parse Multiplicity Mismatch Hypothesis

This paper tests the hypothesis that language models' underestimation of human processing difficulty in syntactic ambiguity arises from their ability to maintain more simultaneous sentence parses than humans, but finds that while reducing parse multiplicity increases predicted garden-path effects, it fails to fully account for the magnitude of human reading time differences.

Original authors: William Timkey, Brian Dillon, Tal Linzen

Published 2026-05-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: William Timkey, Brian Dillon, Tal Linzen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are reading a story, and suddenly you hit a sentence that tricks your brain. This is called a "garden path" sentence.

Here is an example: "The girl fed the lamb was upset because she asked for beef."

When you first read "The girl fed the lamb," your brain naturally assumes the girl is the one doing the feeding. It's the most common way to understand it. But then you hit the word "was." Suddenly, your brain realizes it was wrong! The sentence actually means the girl was fed (someone fed her), and she was upset about the beef. That moment of confusion and re-reading takes time. Psychologists call this a "garden path effect."

The Big Question
Scientists have built powerful computer programs called "Language Models" (like the ones that write this text) that are very good at guessing the next word in a sentence. A theory called "Surprisal Theory" suggests that the harder a word is to predict, the longer it takes humans to read it.

The problem? When scientists tested these computers on garden path sentences, the computers were too calm. They didn't seem surprised enough by the trick. They predicted the confusion would be tiny, but humans actually experience a massive mental "stumble."

The Hypothesis: The "Flashlight" Theory
The authors of this paper wondered: Why are the computers so calm?

They guessed it might be because computers are too smart. When a computer reads "The girl fed the lamb," it might be holding dozens of different interpretations in its head at the same time. It's thinking: "Maybe she's feeding the lamb. Maybe the lamb is feeding her. Maybe it's a different kind of lamb." Because it's looking at all these possibilities at once, the word "was" doesn't feel like a shock; it just feels like one of the many options it was already considering.

Humans, on the other hand, might be like a flashlight in a dark room. We can only shine our light on one or two paths at a time. We commit to the "girl is feeding" idea. When we hit "was," our flashlight is pointing the wrong way, and we have to spin around and find the new path. That spinning takes time and effort.

The authors called this the "Parse Multiplicity Mismatch Hypothesis." In simple terms: Computers are less surprised because they are looking at too many possibilities at once, while humans are forced to pick just one.

The Experiment: Turning Down the Flashlight
To test this, the researchers used a special type of computer model that can be told exactly how many "paths" to look at. They ran the models through the garden path sentences with different settings:

High Multiplicity: The computer looks at 1,000 different interpretations at once (like a super-bright, wide flashlight).
Low Multiplicity: The computer is forced to look at only 1 or 2 interpretations (like a dim, narrow flashlight).

The Results

They were partially right: When they forced the computer to look at fewer paths (narrower flashlight), it did get more surprised. The "stumble" in the computer's prediction got bigger.
But they were mostly wrong: Even when they forced the computer to look at only one path (the wrong one, just like a human), the computer's surprise was still tiny compared to the human reaction.

The computer's "stumble" was still about 40 times smaller than the human stumble.

The Conclusion
The paper concludes that simply making computers "dumber" or forcing them to look at fewer options doesn't fix the problem. The gap between how computers and humans process language is too big to be explained just by how many ideas they can hold in their heads at once.

The authors suggest that human brains might have an extra step that computers don't have: a specific "re-analysis" mechanism. When humans get stuck, we don't just feel a little surprised; we actively tear down our old understanding and rebuild a new one. The current computer models, even when limited, just don't seem to have that "tear down and rebuild" button. They just keep guessing, and they guess too easily.

In a Nutshell
The researchers tried to see if computers were less surprised by tricky sentences because they were "thinking too much" (holding too many ideas). They found that even when they forced the computers to "think less," the computers were still way too calm compared to humans. This means the difference between human and computer reading isn't just about how many ideas we can juggle; it's about something deeper in how we handle being wrong.

Technical Summary: Testing the Parse Multiplicity Mismatch Hypothesis

Problem Statement
Surprisal theory posits that the processing difficulty of a word is determined by its predictability in context (its negative log probability), offering a potential link between human sentence processing and next-word predictions from language models (LMs). While LM surprisals successfully predict reading times in naturalistic text, they systematically underpredict the magnitude of difficulty observed in controlled studies of syntactic ambiguity, particularly in "garden path" sentences. In these sentences, a temporary ambiguity initially favors one interpretation, which is later disambiguated in favor of a dispreferred interpretation, causing significant processing costs (e.g., reading time increases of >300ms).

Previous research indicates that while LMs predict the direction of garden path effects, the magnitude is orders of magnitude smaller than human data. For instance, GPT-2 predicts a garden path effect of approximately 8ms, whereas human data shows effects exceeding 300ms. This paper investigates whether this misalignment arises from differences in computational constraints between humans and LMs. Specifically, it tests the Parse Multiplicity Mismatch Hypothesis: the proposal that LMs can simultaneously maintain a greater number of distinct syntactic interpretations (parses) in memory than humans. If humans are more constrained in their parallelism, they may commit to a single, incorrect interpretation earlier, leading to higher surprisal upon disambiguation compared to LMs that marginalize over many competing parses.

Methodology
To test this hypothesis, the authors utilized Recurrent Neural Network Grammars (RNNGs) (Dyer et al., 2016). Unlike standard autoregressive LMs (e.g., GPT-2) which implicitly learn syntax, RNNGs condition next-word predictions on an explicit syntactic parse of the context. This architecture allows for the direct manipulation of parse multiplicity via word-synchronous beam search.

Model Setup: The authors trained RNNGs on the BabyLM and BLLIP corpora using two parsing strategies: top-down and left-corner.
Manipulating Multiplicity: The key parameter varied was the word beam width ( $k_w$ ), which determines the number of distinct parses retained at each word step.
- $k_w = 1$ : Represents a fully serial parser (maintaining only one parse).
- $k_w \to \infty$ : Represents a fully parallel parser (approximating full marginalization over all parses).
- The study tested $k_w$ values ranging from 1 to 1000.
Surprisal Calculation: Next-word surprisal was computed by marginalizing the next-word probability distribution over the top $k_w$ parses.
Counterfactual Conditions: To establish bounds, the authors introduced two specific conditions:
- FORCED GARDEN PATH: Manually pruning the beam to include only parses consistent with the initially preferred (incorrect) interpretation, creating an upper bound on predicted surprisal.
- FULL-PARALLEL: Manually ensuring the beam always contains at least one parse consistent with the globally correct interpretation, creating a lower bound.
Data and Evaluation: The models were evaluated on the SAP Benchmark (Huang et al., 2024), specifically on three garden path constructions (MV/RR, NP/S, NP/Z). The study compared predicted reading times (derived from surprisal via linear mixed-effect regression on filler sentences) against empirical human reading times.

Key Results

Naturalistic Text: Consistent with prior work, surprisal estimates derived from larger beam widths ( $k_w$ ) provided a better fit to reading times in naturalistic filler sentences. This suggests that, for general text, higher parallelism (more parses) yields more accurate human-like predictions.
Garden Path Effects: Reducing parse multiplicity (lowering $k_w$ ) did increase the magnitude of predicted garden path effects, supporting the intuition that limiting parallelism increases disambiguation costs. The largest effects were observed at intermediate beam widths ( $2 \le k_w \le 10$ ), where the beam was small enough to prune the correct parse but large enough to maintain high-quality parses.
The Gap Persists: Despite increasing the predicted effect size by restricting parallelism, the results failed to close the gap with human data.
- Even at the most restrictive conditions (e.g., $k_w=1$ or the FORCED GARDEN PATH condition where only the incorrect parse was considered), the predicted reading time increases were orders of magnitude smaller than the empirical human effects.
- The FULL-PARALLEL condition (guaranteeing the correct parse is present) did not significantly reduce the underprediction compared to the unmodified beam, indicating the issue is not merely the presence of the correct parse in the beam.

Significance and Claims
The paper concludes that the Parse Multiplicity Mismatch Hypothesis cannot fully reconcile LM-based surprisal with human sentence processing. While limiting the number of simultaneous parses in LMs does increase the predicted difficulty of garden path sentences, it is insufficient to account for the full magnitude of human processing costs.

The authors argue that this finding suggests:

The misalignment is not driven solely by the number of interpretations considered (parse multiplicity).
Even when an LM is forced to consider only the initially preferred (incorrect) interpretation, the resulting surprisal for the disambiguating word remains too low to explain human reading times.
Therefore, explaining the full magnitude of garden path effects likely requires mechanisms beyond single-stage word predictability. The authors suggest that syntactic reanalysis (a second-stage repair process) or other mechanisms not captured by standard surprisal theory are necessary to model human disambiguation difficulty.

The work implies that while LMs may parallel human prediction mechanisms, the specific computational constraints of human memory (in terms of parallelism) are not the primary factor causing the discrepancy in garden path effects. Instead, the nature of the processing cost itself may involve processes (like reanalysis) that are not reducible to word-level surprisal.

More like this