Information-Theoretic Storage Cost in Sentence Comprehension

1. Problem Statement

Real-time sentence comprehension imposes a significant cognitive load on working memory. As comprehenders process input incrementally, they must maintain unresolved syntactic predictions (expectations) about future words.

• The Bottleneck: The accumulation of these unresolved predictions consumes memory resources, creating a processing bottleneck that leads to reading difficulty (e.g., in center-embedded sentences).
• Limitations of Existing Metrics: Traditional measures of this "storage cost" rely on symbolic grammars (e.g., Dependency Locality Theory or DLT). These metrics:
  • Assume discrete, uniform costs for every predicted syntactic element.
  • Depend heavily on specific syntactic theories and manual annotations (e.g., identifying traces or null elements).
  • Fail to capture the probabilistic nature of language and semantic nuances.
• The Gap: There is a need for a continuous, theory-neutral, and grammar-independent measure of storage cost that can be estimated directly from pre-trained neural language models (LMs) to better reflect human processing.

2. Methodology

The authors propose a new metric called Information-Theoretic Storage Cost, grounded in information theory.

A. Theoretical Formalization

Instead of counting syntactic heads, the authors define storage cost as the amount of information a current word carries about future context under uncertainty.

• Predictive Potential ($P_{pred}$): For a target word $w_i$ and a future sequence $w_{[k:N]}$ (given context $w_{[1:k)}$), the predictive potential is defined as the expected contextualized Pointwise Mutual Information (PMI).
  • Mathematically, this is equivalent to the Kullback-Leibler (KL) divergence between the predictive distribution of the future sequence with $w_i$ visible and without $w_i$ visible.
  • Formula: $P_{pred}(w_i; W_{[k:N]}) = E_{w_{[k:N]}|w_{[1:k)}} [ \log \frac{p(w_{[k:N]} | w_{[1:k)})}{p(w_{[k:N]} | w_{[1:k)} \setminus w_i)} ]$
• Information Storage ($InfoStork$): The total storage cost at position $k$ is the sum of the predictive potentials of all preceding words ($w_1$ to $w_{k-1}$).
  • $InfoStork = \sum_{i=1}^{k-1} P_{pred}(w_i; W_{[k:N]})$
• Properties: This measure is continuous (measured in bits), decays monotonically as distance increases (aligning with memory decay theories), and requires no explicit syntactic parsing.

B. Estimation via BERT

Since calculating exact joint probabilities over future sequences is intractable, the authors estimate $P_{pred}$ using BERT (a bidirectional masked language model):

1. Input Construction: For a target word $w_i$ and future region $w_{[k:N]}$, they create two inputs:
   • With $w_i$: The word is visible; future tokens are masked.
   • Without $w_i$: The word is masked; future tokens are masked.
2. Conditional Independence Assumption: To handle multiple masked tokens, they assume conditional independence among masked positions. The total KL divergence is approximated as the sum of KL divergences at each masked token position.
3. Model: They use bert-base-uncased.

3. Key Contributions

1. Formalization: A novel, continuous, and grammar-independent definition of storage cost based on information theory (specifically, contextualized half-pointwise mutual information).
2. Implementation: A practical method to estimate this cost using pre-trained neural language models (BERT) without requiring syntactic annotations.
3. Validation: A comprehensive three-part analysis demonstrating that this measure captures known psycholinguistic phenomena and predicts human reading behavior better than or independently of traditional grammar-based metrics.

4. Results

The study validates the proposal through three analyses:

A. Recovery of Processing Asymmetries

The measure successfully recovers well-known processing difficulties using only distributional statistics:

• Center Embedding vs. Right Branching: Information storage rises sharply in center-embedded structures (e.g., "The reporter who the senator who Mary met attacked...") compared to right-branching variants, mirroring the known cognitive difficulty.
• Object vs. Subject Relative Clauses: The measure predicts significantly higher storage costs for Object Relative Clauses (ORC) compared to Subject Relative Clauses (SRC), peaking at the embedded noun phrase, consistent with behavioral data.

B. Correlation with Grammar-Based Metrics

• The authors tested the correlation between their Information Storage and DLT storage cost using the UD_English-GUM corpus.
• Result: A moderate positive correlation was found ($r \approx 0.34$ to $0.49$). This confirms that the information-theoretic measure captures the core structural intuitions of grammar-based storage but also includes additional semantic and distributional information not captured by discrete syntactic counts.

C. Prediction of Naturalistic Reading Times

The study evaluated the metric on two large-scale datasets (Natural Stories and OneStop) using eye-tracking and self-paced reading data.

• Baseline: Models included word length, position, unigram surprisal, and GPT-2 surprisal.
• Findings:
  • Adding Information Storage significantly improved model fit (measured by $\Delta$ Log-Likelihood) across multiple datasets and reading measures.
  • Independence: Crucially, Information Storage and DLT Storage provided complementary predictive power. Adding Information Storage to a model already containing DLT Storage (and vice versa) yielded significant improvements.
  • Dissociation: In the OneStop dataset, DLT storage was strongest for "Go-Past Duration" (GPD), while Information Storage was strongest for "Total Fixation Duration" (TFD). This suggests Information Storage may better predict regressive eye movements (re-reading), which are often driven by semantic or probabilistic confusion rather than purely structural complexity.

5. Significance and Implications

• Theory-Neutral Framework: The study demonstrates that storage cost can be operationalized without relying on specific syntactic theories (like DLT or CCG), making it applicable across languages and linguistic frameworks.
• Complementary Mechanisms: The results suggest that human sentence comprehension relies on two distinct but complementary mechanisms:
  1. Structural Processing: Captured by grammar-based metrics (handling abstract syntactic dependencies).
  2. Statistical/Probabilistic Processing: Captured by information-theoretic metrics (handling semantic expectations and distributional patterns).
• Future Directions: The authors note limitations, such as the assumption of conditional independence in BERT estimation and the need to explore compressed memory representations (resource-rationality) rather than raw token-level predictions. They suggest future work should investigate cross-linguistic variations, particularly in head-final languages.

In conclusion, the paper establishes Information-Theoretic Storage Cost as a robust, continuous, and empirically valid alternative to traditional grammar-based storage metrics, offering a more nuanced view of the cognitive load involved in real-time language comprehension.