To model human linguistic prediction, make LLMs less superhuman

Technical Summary: "To model human linguistic prediction, make LLMs less superhuman"

Problem Statement
Large Language Models (LLMs) have historically been utilized as computational models of human linguistic prediction, operating on the premise that their ability to predict upcoming words mirrors human cognitive processes. However, a paradox has emerged: as the next-word prediction accuracy of mainstream LLMs has improved, their ability to explain human reading behavior (specifically reading times and eye-tracking data) has declined. The paper argues that this divergence occurs because current LLMs possess "superhuman" predictive capabilities driven by memory capacities that far exceed those of human readers. Specifically, LLMs exhibit superior long-term memory (retaining vast factual knowledge and training examples) and near-perfect short-term memory (faithfully recalling words from distant contexts without temporal decay), leading to predictions that average humans cannot make.

Methodology and Analysis
The paper synthesizes existing empirical literature and re-analyzes the relationship between model performance and human data alignment.

• Literature Synthesis: The authors review the trajectory of studies using LLMs to estimate word predictability. They contrast early findings (where better prediction accuracy correlated with better alignment to human reading times) with recent findings (post-2022) using powerful Transformer-based models trained on massive datasets.
• Data Re-evaluation: The authors examine results from standard corpora, including the Natural Stories Corpus (self-paced reading) and the Dundee Corpus (eye-tracking). They observe that while initial improvements in model accuracy enhanced alignment, further gains in prediction accuracy—driven by increased model scale and training data—resulted in a growing misalignment with human reading times.
• Mechanistic Analysis: The paper dissects the architectural and training differences between humans and LLMs:
  • Long-term Memory: LLMs are trained on orders of magnitude more data (e.g., 15 trillion words for Llama 3 vs. ~100 million for a child by age 12) and retain factual knowledge (e.g., birthplaces of celebrities) and multiword expressions that humans lack or have forgotten.
  • Short-term Memory: Unlike human working memory, which decays over time and is subject to interference, Transformer architectures maintain a "decay-free" store of previous words via attention mechanisms. This allows LLMs to perfectly recall entities mentioned thousands of words prior, a capability that leads to near-perfect prediction of repeated text where humans show processing difficulty.

Key Contributions

1. Identification of "Superhumanness" as a Barrier: The paper posits that the primary obstacle to using LLMs as cognitive models is not their lack of capability, but their excess capability. Their "superhuman" memory allows them to solve prediction tasks that are inherently difficult for humans due to knowledge gaps or memory decay.
2. Diagnosis of Memory Discrepancies: The authors explicitly attribute the misalignment to two specific memory failures:
   • Familiarity Mismatch: LLMs possess factual knowledge and exposure to specific texts (e.g., the Declaration of Independence) that the average human reader does not.
   • Recall Mismatch: LLMs lack the temporal decay and interference effects inherent in human short-term memory, allowing them to resolve syntactic ambiguities and recall repeated entities with perfect accuracy, whereas humans struggle.
3. Proposal for Human-Like Modeling: The paper advocates for a shift in modeling goals from maximizing prediction accuracy to maximizing human-likeness. This involves developing LLMs with human-scale memory constraints.

Results and Evidence

• Divergence Trends: Analysis of regression models fitting LLM probabilities to human reading times shows a non-monotonic relationship. Initial increases in model accuracy improve fit, but beyond a certain threshold, further accuracy gains degrade the fit.
• Memory Manipulation Studies: The paper cites evidence that limiting LLMs' access to context improves alignment. For instance, downweighting high-frequency words in deeply embedded sentences or restricting the attention window to recent words reduces the mismatch between model probabilities and human reading times.
• Training Data Effects: Models trained on smaller, curated datasets (e.g., the BabyLM Challenge) or specialized domains show better alignment with human reading behavior than those trained on massive, general corpora, suggesting that limiting data exposure prevents the acquisition of "superhuman" expertise.

Significance and Future Directions
The paper concludes that for LLMs to serve as valid models of human language comprehension, they must be "made less superhuman." The authors outline several directions to achieve this:

• Training Data Curation: Limiting training data to human-scale amounts (e.g., ~100 million words) to prevent the memorization of obscure facts and reduce the development of perfect short-term memory mechanisms.
• Architectural Changes: Exploring recurrent neural network architectures that force input compression and implement human-like decay, or modifying Transformer attention mechanisms to introduce bottlenecks and decay functions.
• Training Objectives: Shifting from objectives that reward "lexically sharp" predictions (assigning high probability to the single correct word) to objectives that encourage "diffuse" predictions (assigning probability mass to semantically similar words), better reflecting human uncertainty.
• Post-hoc Interventions: Using model editing techniques to "unlearn" specific factual knowledge that humans typically lack.
• New Experimental Paradigms: The authors call for targeted human experiments to benchmark progress. These experiments should systematically manipulate long-term knowledge (via questionnaires) and short-term memory loads (via list recall and intervening text) to establish a causal link between memory constraints and reading behavior.

The paper emphasizes that closing the gap between LLMs and humans requires not just better models, but models that are constrained by human limitations in memory and knowledge.