💬 NLP

Eyettention II: A Dual-Sequence Architecture for Modeling Fixation Location, Within-Word Landing Position, and Fixation Duration in Reading

To address the scarcity of eye-tracking data, the paper introduces Eyettention II, a lightweight, end-to-end deep learning model that generates realistic, multi-attribute reading scanpaths by aligning with cognitive theories and outperforming state-of-the-art models in predicting human-like gaze behavior.

Original authors: Shuwen Deng, Cui Ding, David R. Reich, Paul Prasse, Lena A. Jäger

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shuwen Deng, Cui Ding, David R. Reich, Paul Prasse, Lena A. Jäger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are reading a book. Your eyes don't glide smoothly across the page like a car on a highway; instead, they jump in little bursts called "fixations," landing on specific words, sometimes skipping ahead, sometimes jumping back to re-read. This pattern of jumps is called a scanpath.

For decades, scientists have tried to build computer models that can predict exactly where your eyes will land and how long they will stay there. The problem is that collecting real eye-tracking data is expensive and slow—it's like trying to film a movie where every actor has to wear a special camera on their head. Because there isn't enough "real" footage, it's hard to train computers to be good at predicting these jumps.

Enter Eyettention II. Think of this new model as a highly skilled simulator or a virtual eye-tracker. It's a piece of software that can generate realistic "fake" eye movement data for any text, complete with where the eye lands, exactly where on the word it stops, and how long it pauses.

Here is a breakdown of how it works and what the paper found, using simple analogies:

1. The "Dual-Sequence" Puzzle

Most computer models treat reading like a simple list: Word 1, Word 2, Word 3. But human reading is messy. We skip words, jump back, and stare at long words longer than short ones.

The authors describe Eyettention II as a dual-sequence architect. Imagine two conveyor belts running side-by-side:

Belt A (The Text): The words in the order they appear in the sentence.
Belt B (The Time): The order in which your eyes actually look at things.

Because your eyes don't always follow the text perfectly (you might jump back to the start), these two belts get out of sync. Eyettention II is special because it has a "traffic controller" (a cross-attention mechanism) that watches both belts simultaneously. It knows that just because the text says "Word 5" is next, your eyes might actually be jumping back to "Word 2." This allows it to mimic the chaotic, non-linear way humans actually read.

2. What Does It Predict?

Older models were like a GPS that only told you which city you were in. Eyettention II is a high-definition GPS that tells you:

The City (Word Index): Which word is being looked at?
The Street Address (Landing Position): Exactly where on that word does the eye land? (e.g., the beginning, the middle, or the end).
The Dwell Time (Duration): How long does the eye stay there?

It generates these three things in a chain, one after another, just like a human reading in real-time.

3. The "Personal Trainer" Mode

The paper also introduces a special version called Eyettention IIreader.
Imagine you have a generic fitness coach who knows how the average person runs. But what if you want a coach who knows your specific running style? This version of the model can be "tuned" to a specific person (or a group of people). If you feed it data from a specific reader, it learns their unique habits—maybe they skip short words more often, or they stare longer at difficult words—and it can then simulate that specific person's eye movements.

4. How Good Is It?

The researchers tested this model on English and Chinese texts (two very different languages). They compared it to:

Older AI models: The new model won, predicting eye movements more accurately.
Classic "Cognitive" models: These are older, rule-based models built by psychologists to mimic human thought. Eyettention II beat them too, showing that a data-driven approach can capture complex patterns that simple rules miss.
Real Humans: Interestingly, the model's "fake" eye movements were sometimes more similar to each other than real humans were to each other. This is because real humans are messy and vary wildly, while the model finds the "average" human pattern.

5. Why Do We Need a Fake Eye-Tracker?

The paper highlights three main reasons to use this simulator, all related to saving time and money:

The "Pilot Test" (Experiment Planning): Before a scientist runs a real experiment with 50 people, they can use the simulator to "run" the experiment 1,000 times virtually. This helps them figure out if their test materials are good and how many people they actually need to recruit. It's like a flight simulator for researchers.
The "Data Augmenter" (Training AI): If you want to build an AI that understands reading, you need tons of eye-tracking data. Since real data is scarce, you can use Eyettention II to generate millions of "fake" eye-tracking records to train your AI, making it smarter without needing to strap cameras to thousands of real people.
The "What-If" Tool: Researchers can use it to see how different text layouts or languages might affect reading, without having to set up expensive lab equipment.

6. A Surprising Discovery: "Bigger Isn't Better"

The researchers tested the model using different sizes of "brain" (Language Models) to understand the text. They found a counter-intuitive result: Smaller models worked better.

They tried using massive, complex AI models (like the ones that power advanced chatbots) to understand the text.
They also tried smaller, simpler models.
Result: The smaller models actually predicted human eye movements better. It seems that for reading, you don't need a super-complex brain that understands deep philosophy; you need a model that understands the immediate, surface-level structure of the sentence.

Summary

Eyettention II is a lightweight, efficient computer program that acts as a "virtual eye." It doesn't just guess which word you'll read next; it predicts exactly where your eye will land on that word and how long it will stare, mimicking human behavior with high accuracy. It solves the problem of not having enough real eye-tracking data by generating high-quality "fake" data, which helps scientists plan better experiments and train smarter AI systems.

Technical Summary: Eyettention II

Problem Statement
Eye-tracking-while-reading data offers critical insights into cognitive processes and text properties, driving applications in natural language processing (NLP), such as enhancing language models (LMs) and inferring reader characteristics. However, the field faces a significant bottleneck: data scarcity. Collecting large-scale eye-tracking datasets is resource-intensive, limiting the training of data-driven models and hindering the replication of psycholinguistic experiments. Existing computational models often rely on heuristic, rule-based approaches with few parameters to ensure interpretability, which constrains their ability to capture complex patterns and generalize across different populations, languages, and experimental setups. Furthermore, previous machine learning approaches often focused on either spatial aspects (fixation location) or temporal aspects (aggregated fixation duration) in isolation, failing to generate complete, sequential scanpaths that capture the full dynamics of human reading.

Methodology
The authors propose Eyettention II, an end-to-end trained, lightweight deep-learning model designed to generate realistic scanpaths consisting of a complete set of fixation attributes in chronological order.

Dual-Sequence Architecture: The core innovation addresses the "dual-sequence" nature of reading data: the linguistic sequence (words ordered by grammar) and the temporal sequence (fixations ordered by time). Unlike standard sequence models or encoder-decoder architectures that align a single input with a single output, Eyettention II employs a dual-sequence bi-encoder structure. It utilizes a Word-Sequence Encoder (processing the stimulus text) and a Fixation-Sequence Encoder (processing the history of fixations).
Cross-Sequence Attention: The model aligns these two sequences using a cross-attention mechanism with a local window. This simulates the human visual field by attending only to words surrounding the currently fixated word, rather than the entire sentence, preserving autoregressivity in the temporal dimension while leveraging rich linguistic context.
Prediction Modules: The model predicts three attributes sequentially for each fixation:
1. Word Index ( $k$ ): A multi-class classification task predicting which word is fixated next.
2. Landing Position ( $l$ ): A regression task predicting the normalized horizontal offset within the word (0 to 1).
3. Fixation Duration ( $d$ ): A regression task predicting the time spent on the fixation.
Reader-Specific Variant: The authors introduce Eyettention II $_{reader}$ , which incorporates a trainable reader identifier embedding. This allows the model to learn and reproduce characteristic reading patterns of specific individuals or groups.
Training: The model is optimized using a combined loss function consisting of negative log-likelihood (NLL) for word index and mean-squared-error (MSE) for landing position and duration. It utilizes pre-trained Large Language Models (LLMs) like BERT or RoBERTa as the core of the Word-Sequence Encoder, which are frozen during training, followed by a BiLSTM layer.

Key Contributions

Comprehensive Scanpath Generation: Eyettention II is the first model to simultaneously predict fixation location, within-word landing position, and fixation duration in a single, end-to-end framework, moving beyond the limitations of predicting only fixation locations or aggregated durations.
State-of-the-Art Performance: The model achieves superior performance in both within-dataset and cross-dataset evaluations compared to existing cognitive models (E-Z Reader, SWIFT) and machine learning baselines.
Psycholinguistic Validity: The model successfully mirrors human-like gaze behavior, reproducing key psycholinguistic phenomena such as the effects of lexical frequency, word length, and surprisal on fixation durations and regressions.
Reader Adaptability: The introduction of the reader-specific variant demonstrates that the model can effectively simulate individual reading styles, a crucial capability for personalized applications.
Utility for Experimental Planning: The authors demonstrate the model's utility in conducting statistical power analyses and piloting psycholinguistic experiments, showing that simulated data can provide realistic effect size estimates, particularly when combined with a pre-training and fine-tuning strategy for small datasets.

Results

Quantitative Performance: On both English (CELER) and Chinese (BSC) datasets, Eyettention II outperformed reference models (including E-Z Reader, SWIFT, and previous Eyettention versions) across all metrics: word index prediction (NLL), landing position (MAE), fixation duration (MAE), and overall scanpath similarity (MultiMatch).
Generalization: In cross-dataset evaluations (training on CELER, testing on ZuCo), the model showed significant performance drops when trained from scratch on the target data. However, a pre-training and fine-tuning strategy (pre-training on a large source dataset and fine-tuning on a small target dataset) effectively restored performance, enabling accurate modeling even with limited target data.
Psycholinguistic Alignment: Bayesian mixed-effects analyses confirmed that Eyettention II captures the direction and magnitude of effects for word length, frequency, and surprisal similarly to human readers, outperforming traditional cognitive models in many measures.
Model Architecture Insights:
- LM Size: Contrary to general scaling laws, smaller LMs (e.g., BERT/RoBERTa 110M) provided better linguistic features for scanpath prediction than larger models (up to 7B parameters).
- Layer Depth: Lower BERT layers were more predictive of spatial aspects (word index, landing position), while upper layers were more influential for temporal aspects (fixation duration).
Decoding Strategies: The choice of decoding strategy (Greedy, Temperature Sampling, Nucleus Sampling) allows for a trade-off between generating "average" high-probability scanpaths and more diverse, variable scanpaths. Nucleus sampling ( $p=0.7$ ) and low-temperature sampling ( $t=0.04$ ) yielded the best results for different datasets.

Significance and Claims
The paper claims that Eyettention II represents a significant advancement in modeling human reading behavior by bridging the gap between cognitive theory and data-driven deep learning. Its primary significance lies in its ability to generate realistic, complete scanpaths with limited data requirements, making it a versatile tool for:

NLP Applications: Pre-training or fine-tuning gaze-augmented language models and inferring reader characteristics without requiring real-time eye-tracking data for every input.
Psycholinguistic Research: Facilitating the piloting of experimental materials and conducting statistical power analyses to mitigate the replication crisis caused by small sample sizes.
Theoretical Insight: Providing a data-driven tool to investigate which linguistic and visual features guide human reading behavior, uncovering patterns not explicitly encoded in theoretical cognitive models.

The authors maintain a modest stance, acknowledging that while the model captures key phenomena, it is not fully interpretable like rule-based cognitive models. They note that further refinement is needed to better match human variability in binary measures (like regression probabilities) and to improve performance on Chinese datasets compared to English. The work emphasizes that the model is a tool for generating hypotheses and simulating data, rather than a complete replacement for human experimentation.