Clozing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze Surprisal

Technical Summary: Closing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze Surprisal

Problem Statement
The predictability of a word in context is a fundamental metric for understanding language comprehension and processing effort. Traditionally, this has been quantified using the cloze task, where human participants complete incomplete sentences. More recently, Language Model (LM) probabilities (specifically surprisal, defined as negative log probability) have emerged as an alternative. While recent studies indicate that LM surprisal generally outperforms cloze surprisal as a predictor of reading times (RTs), simply replacing the cloze task with LMs based on fit metrics is scientifically premature. Different predictors can lead to divergent conclusions regarding the cognitive mechanisms of prediction (e.g., whether effects are facilitatory or costly). The core problem addressed is identifying why LM probabilities provide a better fit to human data: is it because LMs capture genuine human-like prediction mechanisms, or because they possess technical advantages (e.g., higher resolution, semantic granularity) that the cloze task lacks?

Methodology
The authors conducted a three-part experimental study using four English reading time datasets (BK21 SPR, Provo ET, UCL SPR/ET) with available cloze response norms.

1. Experiment 1: Baseline Comparison and Optimization

   • Goal: Re-establish the superiority of LM surprisal over cloze surprisal and determine the optimal mathematical transformation for cloze probabilities.
   • Method: The authors fitted Linear Mixed-Effects (LME) regression models to predict word-by-word reading times. They compared models using:
     • LM Surprisal: Calculated using GPT-2 (small).
     • Cloze Surprisal: Derived from human responses, subject to various smoothing techniques (add-one smoothing with varying $V$) and functional transformations (raw probability, raw surprisal, and power transforms of surprisal).
   • Analysis: Models were evaluated using 10-fold cross-validation log likelihood. The authors tested whether LM surprisal subsumes the predictive power of cloze surprisal and vice versa.

2. Experiment 2: Hypothesis-Driven Manipulations

   • Goal: Isolate the specific factors contributing to LM superiority by manipulating GPT-2 probabilities to match the limitations of the cloze task.
   • Hypothesis 1 (Resolution): LMs suffer less from low resolution (sparse data) than cloze tasks.
     • Manipulation: GPT-2 probabilities were resampled to match the exact number of samples ($N$) available in the cloze norms for each context, effectively lowering the LM's resolution to that of the human data.
   • Hypothesis 2 (Semantics): LMs distinguish between semantically similar words (e.g., couch vs. sofa), whereas human predictions may group them.
     • Manipulation: GPT-2 vocabulary was clustered using k-means on token embeddings. The probability of a word was replaced by the sum of probabilities of all words in its semantic cluster, forcing semantically similar words to share the same probability mass.
   • Hypothesis 3 (Frequency): LMs assign probabilities to low-frequency continuations that humans rarely produce.
     • Manipulation: GPT-2's vocabulary was restricted to high-frequency tokens only; probabilities for low-frequency tokens were zeroed out and the distribution renormalized.

3. Experiment 3: Combining Cloze and LM Probabilities

   • Goal: Explore whether integrating similarity information into cloze data (Similarity-Adjusted or SA Surprisal) can bridge the performance gap.
   • Method: The authors calculated SA surprisal for both cloze responses and GPT-2 samples. This metric weights the probability of alternative completions by their semantic distance to the observed word. This was compared against standard "count-and-divide" cloze surprisal.

Key Results

• Experiment 1: LM surprisal (GPT-2) consistently outperformed cloze surprisal across most reading time measures (4 out of 6), subsuming the predictive power of cloze data. The optimal transformation for cloze data was found to be the squared surprisal ($S(w_t)^2$) with a smoothing factor of $V=200$, challenging the assumption of linearity in cloze effects.
• Experiment 2: All three hypotheses were supported. When GPT-2 probabilities were manipulated to match the limitations of the cloze task:
  • Resolution: Reducing GPT-2 resolution to match cloze sample sizes significantly decreased its predictive fit.
  • Semantics: Clustering semantically similar words (removing fine-grained distinctions) significantly decreased fit.
  • Frequency: Restricting probabilities to high-frequency words significantly decreased fit.
  • Conclusion: The advantage of LM surprisal stems from its ability to provide high-resolution estimates, distinguish between semantically similar words, and accurately model low-frequency continuations.
• Experiment 3: Similarity-Adjusted (SA) surprisal, whether derived from cloze responses or GPT-2 samples, performed poorly compared to standard surprisal measures. This suggests that simply weighting alternatives by semantic distance does not resolve the gap, and that the "count-and-divide" method remains a reasonable baseline for cloze data.

Significance and Claims
The paper argues that the superior performance of LM surprisal is not necessarily evidence that LMs perfectly model human prediction mechanisms. Instead, the advantage arises because LMs possess technical capabilities (high resolution, semantic granularity, and coverage of low-frequency items) that the traditional cloze task lacks due to its offline, production-based nature and limited sample sizes.

The authors claim that:

1. Cloze studies need higher resolution: Simply collecting more human responses (beyond the typical ~40-90 per sentence) might improve the predictive power of cloze data, potentially narrowing the gap with LMs.
2. Human prediction sensitivity is an open question: It remains to be determined whether human readers actually make the fine-grained distinctions between semantically similar or low-frequency words that LMs make. Future experiments should test if human expectations are as sensitive to these distinctions.
3. Methodological caution: Researchers should not replace the cloze task with LMs solely based on fit metrics. Understanding the source of the fit difference is crucial for drawing valid scientific conclusions about the role of prediction in language comprehension. The authors advocate for developing alternative tasks (e.g., timed cloze or maze tasks) that better capture real-time probabilistic intuitions rather than offline production.