💬 NLP

Probing for Reading Times

This study demonstrates that early-layer representations in language models effectively predict early-pass human reading times across five languages, outperforming scalar surprisal, while scalar surprisal remains superior for late-pass measures, highlighting a functional alignment between model depth and the temporal stages of human reading.

Original authors: Eleftheria Tsipidi, Samuel Kiegeland, Francesco Ignazio Re, Tianyang Xu, Mario Giulianelli, Karolina Stanczak, Ryan Cotterell

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eleftheria Tsipidi, Samuel Kiegeland, Francesco Ignazio Re, Tianyang Xu, Mario Giulianelli, Karolina Stanczak, Ryan Cotterell

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a movie of someone reading a book. You can see their eyes stop (or "fixate") on certain words for a split second longer than others. In the world of reading research, these tiny pauses are like heartbeats of the brain. A longer pause usually means the brain is working harder to understand that specific word or phrase.

For a long time, scientists have tried to predict these pauses using a simple math trick called "Surprisal." Think of Surprisal as a "shock meter." If a sentence says, "The cat sat on the mat," the word "mat" has a low shock meter because it's expected. If the sentence says, "The cat sat on the... toaster," the shock meter spikes. The theory was: The higher the shock, the longer the pause.

But in this new paper, the researchers asked a bigger question: "Is the shock meter the whole story? Or does the computer's brain (the Language Model) have a more detailed map of how humans read?"

Here is the breakdown of their discovery, using some everyday analogies.

1. The "Layer Cake" of the AI Brain

Modern AI models (like the ones used in this study) aren't just one big brain; they are like a multi-layered cake.

The Bottom Layers (Early Layers): These are like the "appetizers." They handle the basics: recognizing letters, sounds, and simple word shapes.
The Middle Layers: These start mixing things up, looking at grammar and sentence structure.
The Top Layers (Late Layers): These are the "main course." They understand the deep meaning, the context, and the complex ideas.

The researchers peeled back the cake, layer by layer, to see which part of the AI's brain best predicted how long a human would pause while reading.

2. The Big Discovery: It Depends on When You Look

The results were fascinating because they showed that different layers of the AI predict different stages of human reading.

The "First Glance" (Early Layers)

When a human first looks at a word, their eyes make a quick decision: "Do I know this word? Does it look right?"

The Finding: The bottom layers of the AI were the best at predicting these quick, initial pauses.
The Analogy: Imagine you are walking into a grocery store. The bottom layers of the AI are like your eyes scanning the aisle to see if you recognize the brand of cereal. They are fast, structural, and focused on the "look" of the word. The AI's early layers capture this "first impression" better than the simple "shock meter" (Surprisal) ever could.

The "Deep Dive" (Top Layers & Surprisal)

Sometimes, a reader gets stuck. They might re-read a sentence to figure out the grammar or the deep meaning. This takes longer.

The Finding: For these long, deep pauses, the simple "shock meter" (Surprisal) actually worked best, often beating the complex AI layers.
The Analogy: This is like the moment you realize, "Wait, if the cat sat on the toaster, why isn't it on fire?" You have to stop and think about the logic. The simple math of "how unexpected is this?" is surprisingly good at capturing this moment of confusion. The deep, complex layers of the AI didn't add much extra value here; the simple surprise was enough.

3. The "Swiss Army Knife" vs. The "Specialist"

The researchers also found that the "best tool" changes depending on the language.

In English, the AI's early layers were great specialists for the first glance.
In languages like Greek, Hebrew, or Turkish, the simple "shock meter" often did just as well as the fancy AI layers.
The Takeaway: There is no single "magic bullet." Just like you wouldn't use a sledgehammer to crack a nut, you can't use one single AI measurement to predict every reading pause in every language.

4. The "Best of Both Worlds"

The most exciting part? When they combined the AI's early layers (the "first glance" specialist) with the Surprisal (the "shock meter"), the prediction got even better.

The Analogy: It's like having a team. One person is great at spotting the shape of the word (Early Layers), and another is great at calculating how weird the sentence is (Surprisal). When they work together, they can predict exactly when a human will pause, even better than either could alone.

Summary

This paper is a reminder that human reading is a two-step dance:

Step 1: A quick, visual check (Is this a word I know?). The AI's early layers are the best at predicting this.
Step 2: A slower, logical check (Does this make sense?). The simple Surprisal math is surprisingly good at predicting this.

The authors show that while AI models are getting smarter, they aren't just "black boxes." By looking inside their "layers," we can see a functional map that mirrors how our own brains process language, from the first split-second glance to the deep, thoughtful pause.

1. Problem Statement

While it is well-established that neural language models (LMs) encode rich linguistic information (syntax, morphology, semantics) in their internal representations, it remains unclear whether these representations capture cognitive signals related to human reading processing. Specifically, the field has largely relied on surprisal (the negative log-probability of a token/word) as the primary scalar predictor for human reading times. However, recent evidence suggests that larger models with lower perplexity sometimes fit human reading data worse, and that scalar compression (reducing high-dimensional states to a single probability) may discard psychometrically relevant information.

The core research question is: Do the high-dimensional internal representations of language models at different layers predict human reading times better than scalar metrics like surprisal, and does this vary by reading stage (early vs. late processing) and language?

2. Methodology

Data and Models

Corpora: The study utilizes two major eye-tracking corpora:
- Provo: English-only data (84 native readers).
- MECO (Multilingual Eye Movement Corpus): Data from 13 languages; the authors selected English, Greek, Hebrew, Russian, and Turkish to cover diverse language families.
Reading Metrics: Three standard eye-tracking measures were analyzed:
- First Fixation Duration (FFD): Early-pass measure (initial lexical access).
- Gaze Duration (GD): Early-pass measure (sum of fixations before leaving the word).
- Total Reading Time (TRT): Late-pass measure (includes regressions, reflecting integration and discourse processing).
Models:
- mGPT: A 1.3B parameter multilingual model (24 layers).
- GPT-2 Small: English monolingual (12 layers).
- cosmosGPT: Turkish monolingual (12 layers).

Predictors Compared

The authors compared representation-based predictors against scalar baselines using regularized linear regression:

Scalar Baselines:
- Surprisal: Standard next-token probability from the final layer.
- Information Value: Expected distance between the observed next unit and alternative continuations (based on cosine distance of representations).
- Logit-Lens Surprisal: Surprisal computed by projecting intermediate layer representations through the final output head (softmax).
Representation-Based Predictors:
- Raw hidden state vectors ( $h_\ell$ ) extracted from every layer ( $\ell = 1 \dots L$ ) of the models.
- For multi-token words, representations were aggregated via mean-pooling over the constituent tokens.

Experimental Setup

Modeling: Regularized linear regression (Ridge and LASSO) was used to predict reading times from features.
Evaluation: 10-fold cross-validation was performed. Performance was measured via Mean Squared Error (MSE) on the validation set.
Controls:
- Permutation Test: Reading times were randomly shuffled to ensure predictors weren't capturing random noise.
- Linear Mixed-Effects Models (LMMs): Used to account for subject and document variability, with representations reduced to 25 principal components (PCA) to handle dimensionality.
- Combined Settings: Tested if adding scalar predictors (surprisal, info value) to representations improved performance.

3. Key Results

Layer-Specific Performance (The "Depth-Time" Alignment)

Early-Stage Processing (FFD & GD): Representations from early layers (typically layers 1–10) significantly outperformed scalar surprisal in predicting First Fixation Duration and Gaze Duration. This suggests that early layers capture low-level structural and lexical features relevant to initial word recognition.
Late-Stage Processing (TRT): For Total Reading Time, scalar surprisal (and logit-lens surprisal) often remained superior or competitive with high-dimensional representations, despite being a much more compressed signal.
Pattern: Predictive power for representations follows a U-shaped curve: high in early layers, dipping in intermediate layers, and recovering slightly at the final layer.

Cross-Lingual Variation

The "best" predictor is highly dependent on the language and the specific eye-tracking measure.
English: Early-layer representations were consistently strong for early-pass measures.
Other Languages (Greek, Hebrew, Russian, Turkish): Scalar predictors were frequently as strong as or stronger than representations, depending on the metric. For example, in Turkish and Russian, the combination of representation + surprisal did not always yield the best results for FFD/GD, unlike in English.

Combined Predictors

Combining representations + surprisal frequently improved performance over representations alone, particularly for early-pass measures.
However, gains over scalar baselines were inconsistent. In some cases (e.g., Turkish FFD), adding representations to surprisal did not improve performance.

Information Value & Logit Lens

Information Value: Showed higher predictive power in early-to-intermediate layers.
Logit-Lens Surprisal: Performed best in later layers, contrasting with the raw representations which peaked in early layers. This highlights a divergence in how different extraction methods utilize model depth.

4. Key Contributions

Functional Alignment Hypothesis: The paper provides empirical evidence for a functional alignment between model depth and the temporal stages of human reading. Early layers correspond to early-pass processing (lexical access), while later layers/scalar outputs correspond to late-pass processing (integration).
Beyond Surprisal: It demonstrates that high-dimensional internal representations contain psychometric information that scalar surprisal misses, specifically for the initial stages of reading.
Cross-Linguistic Nuance: It challenges the universality of surprisal as the "gold standard" predictor, showing that the optimal predictor varies significantly across typologically distinct languages (e.g., agglutinative Turkish vs. inflectional Russian).
Methodological Rigor: The study employs a comprehensive suite of controls, including permutation tests, LMMs, and comparisons across multiple model architectures and languages.

5. Significance and Implications

Cognitive Modeling: The findings suggest that language models are not just "black boxes" predicting the next token; their internal state evolution mirrors the cognitive stages of human reading. This supports the use of LMs as cognitive models for psycholinguistics.
Model Interpretability: The results imply that probing for cognitive signals requires looking at specific layers rather than just the final output. Early layers are crucial for understanding immediate processing effort, while later layers capture integrative effort.
Future Directions: The paper suggests that future work should explore non-linear probing methods (e.g., Kernel PCA) to see if reading-time-relevant structure exists but is not linearly accessible. It also calls for testing these hypotheses on larger models and other modalities (e.g., EEG/fMRI).

In conclusion, the paper establishes that language model representations are superior to scalar surprisal for predicting early-stage reading effort, but surprisal remains a robust predictor for late-stage integration, with significant variations across languages. This refines our understanding of how neural networks encode human-like processing dynamics.