Human-like fleeting memory improves language learning but impairs reading time prediction in transformer language models

Technical Summary: Human-like Fleeting Memory Improves Language Learning but Impairs Reading Time Prediction in Transformer Language Models

Problem Statement
Cognitive science has long hypothesized that the fleeting nature of human working memory—specifically the rapid loss of exact wordforms—paradoxically aids language acquisition. This "less is more" hypothesis suggests that memory limitations force learners to focus on local regularities and develop abstract representations (e.g., chunking, syntactic structure) rather than relying on surface-level statistical memorization. Classic connectionist work by Elman (1993) supported this, showing that limiting input windows improved learning in recurrent neural networks. However, the rise of Transformer models, which possess perfect memory within their context windows and lack inherent recency biases, challenges this view. Transformers learn language effectively despite these "superhuman" memory capabilities. This paper investigates whether reintroducing human-like memory limitations (fleeting memory) actually benefits language learning in modern Transformers, and whether such architectural constraints improve the models' ability to predict human reading behavior.

Methodology
The authors conducted tightly controlled experiments using Transformer language models (GPT-2 architecture) trained on the BabyLM dataset, which simulates developmentally realistic data regimes (10M and 100M tokens).

• Architecture Modification: The authors implemented "fleeting memory" by modifying the self-attention mechanism. Instead of uniform attention weights, they applied a fixed, non-trainable recency bias (a power-law decay function) to the attention scores after the softmax operation. This bias reduces the retention of tokens based on their distance from the current word.
• Echoic Memory Buffer: Recognizing that naive decay (immediate loss from the first token) disrupted local dependencies, the authors introduced an "echoic memory buffer." This ensures perfect retention for the initial 3–7 words (approx. 5–10 tokens) before the power-law decay begins, mimicking human sensory memory.
• Experimental Design: To isolate the effect of memory architecture, the authors trained models with identical random seeds (controlling for weight initialization and data sampling order) under two conditions: Perfect Memory (standard Transformer) and Fleeting Memory (with varying decay strengths and buffer sizes).
• Evaluation Metrics:
  1. Language Learning: Measured via validation loss (cross-entropy) and targeted syntactic evaluation using the BLiMP benchmark (linguistic minimal pairs).
  2. Behavioral Alignment: Measured by the model's ability to predict human reading times (surprisal-based prediction) using linear mixed-effects regression on the Natural Stories and Dundee corpora.
  3. Statistical Rigor: Analyses used bootstrap t-tests across 10–12 seeds per condition to ensure results were not artifacts of stochasticity.

Key Results

1. Naive Decay Impairs Learning: A "naive" implementation of fleeting memory (immediate decay without a buffer) consistently degraded language modeling performance, increasing validation loss.
2. Fleeting Memory + Buffer Improves Learning: When combined with an echoic memory buffer, fleeting memory consistently improved language learning. Models with this architecture achieved lower validation loss and higher BLiMP scores compared to perfect memory controls across both 10M and 100M training sets. The improvements were statistically significant and robust across different hyperparameter settings.
3. Syntactic Knowledge Gains: The improvements extended to abstract syntactic phenomena, particularly those relying on local dependencies (e.g., Subject-Verb Agreement, Anaphor Agreement).
4. Impaired Reading Time Prediction: Contrary to the expectation that better language models predict human behavior better, the fleeting memory models performed worse at predicting human reading times. This impairment was observed across both training set sizes and datasets (Natural Stories and Dundee).
5. Dissociation Mechanism: The authors ruled out standard explanations for the "better model, worse behavior" paradox. The impairment was not due to superhuman training scales (as the data was human-scale) nor solely due to the memorization of low-frequency words (though the error did concentrate on rare words, the specific signature of underprediction associated with memorization was absent).

Significance and Claims
The paper claims to provide empirical support for the cognitive hypothesis that memory limitations serve as a beneficial inductive bias for language learning in neural networks, even within the Transformer architecture. By encoding the statistical prior that linguistic dependencies are often local, fleeting memory improves sample efficiency and the acquisition of syntactic structure.

However, the authors highlight a critical dissociation: architectural constraints that make a model more "human-like" in its learning process do not necessarily make it more "human-like" in its prediction of human behavior (reading times). The results suggest that multiple distinct mechanisms can cause a divergence between language modeling competence and behavioral alignment. The authors modestly note that these findings are specific to human-scale data regimes; at contemporary pretraining scales (billions of tokens), the benefits of such constraints may diminish as models can learn local biases directly from data. The work underscores that improving a model's internal learning efficiency does not guarantee a better fit to human processing data.