💬 NLP

Eye Tracking Based Cognitive Evaluation of Automatic Readability Assessment Methods

This paper introduces an eye-tracking-based cognitive evaluation framework that reveals traditional readability formulas, NLP methods, commercial systems, and frontier LLMs are poor predictors of real-time reading ease compared to psycholinguistic word properties, highlighting the need for new cognitively driven scoring approaches.

Original authors: Keren Gruteke Klein, Shachar Frenkel, Omer Shubi, Yevgeni Berzak

Published 2026-07-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Keren Gruteke Klein, Shachar Frenkel, Omer Shubi, Yevgeni Berzak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out how easy a book is to read. For over a hundred years, scientists and teachers have tried to build "readability formulas"—mathematical recipes that look at a text and spit out a grade level, like "this is a 5th-grade text" or "this is a 12th-grade text." They usually do this by counting things like how long the sentences are or how many big words are in them. But here's the tricky part: just because a text has short sentences doesn't mean a human actually feels like it's easy to read while they are reading it. To really know if a text is easy, you need to see how a person's brain is working in real-time. This is where "eye tracking" comes in. It's like putting a tiny camera on a person's eye to see exactly where they look, how long they stare at a word, and if they have to look back (regress) because they got confused. It's the difference between guessing how fast a car is going by looking at the engine size versus actually driving it and feeling the speed.

This paper, titled "Eye Tracking Based Cognitive Evaluation of Automatic Readability Assessment Methods," is a big reality check for the world of text analysis. The researchers, led by Keren Gruteke Klein and her team at the Technion in Israel, decided to test the most popular ways we currently judge text difficulty. They gathered a massive group of 638 adult readers—some native English speakers and some who learned English as a second language—and had them read news articles on a computer while their eyes were tracked at a super-fast speed of 1,000 times per second. They used a clever trick: they took the same news stories and had human teachers rewrite them to be simpler. This allowed the researchers to compare the "hard" version and the "easy" version of the exact same content, isolating the writing style from the topic.

The team then asked a simple question: Which computer program is the best at predicting how much easier the "easy" version actually felt to the readers? They tested everything from old-school math formulas (like the Flesch Reading Ease score) to modern AI systems, commercial tools used in schools, and frontier Large Language Models (LLMs). Notably, the study evaluated models like GPT-4o and other cutting-edge systems (including versions labeled with future dates like 2025 in the paper) using specific prompts that asked the AI to assign a grade level or a difficulty score from 1 to 100. They compared these high-tech tools against some very simple, old-school measurements from psychology, like how often a word is used in the language or how surprising a word is to see next.

The results were a bit of a shock. Despite all the fancy technology and the incredible power of modern AI, the popular readability tools were terrible at predicting how easy a text actually felt to read in real-time. The old formulas, the commercial school tools, and even the cutting-edge AI models all failed to match the readers' eye movements. In fact, the best predictors turned out to be the simplest things: how long a word is, how common it is, and a measure called "surprisal" (which is basically a math way of saying "how unexpected is this word?"). The study suggests that while our current tools are great at guessing what grade level a text belongs to, they are missing the mark on the actual human experience of reading. The authors conclude that we need to stop relying on these old methods for high-stakes decisions and start building new systems that are actually driven by how our brains process words in the moment.

Technical Summary: Eye Tracking Based Cognitive Evaluation of Automatic Readability Assessment Methods

Problem Statement

Automatic Readability Assessment (ARA) has been a research domain for over a century, widely utilized in education, healthcare, and media. Historically, the development and evaluation of ARA methods have relied on two primary offline behavioral data sources: performance on reading comprehension tests and human ratings of text readability levels. However, these approaches capture offline signals that do not directly measure a fundamental aspect of readability: real-time reading ease. While reading ease was proposed early on as a key construct (e.g., Dale and Chall 1949; Kintsch and Vipond 1979), it has remained understudied in ARA due to the scarcity of behavioral data from the reading process. Furthermore, existing evaluation methods often suffer from the entanglement of text difficulty arising from topical variation versus linguistic complexity.

Methodology

The authors introduce a cognitive evaluation framework that utilizes eye-tracking data to quantify the ability of readability scoring methods to account for reading ease, specifically controlling for content variation.

Data Collection

The study utilizes a combined dataset, OneStopL1&L2, comprising over 3.7 million word tokens from 638 adult participants (360 L1 English speakers and 278 L2 speakers with diverse native backgrounds).

Textual Materials: The dataset uses a parallel corpus from the OneStopQA dataset, consisting of 30 Guardian news articles. Each article exists in an original "Advanced" version and a human-simplified "Elementary" version, created by experienced language teachers using an intuitive simplification method.
Experimental Design: Participants read paragraphs in either an information-seeking or comprehension regime. The design ensures that each participant reads both original and simplified versions of the same content, allowing for the isolation of simplification effects.
Eye Tracking: Data was collected using an EyeLink 1000 Plus at 1000 Hz.

Evaluation Framework

The core metric is reading facilitation, defined as the change in reading ease measures between the original and simplified versions of a text.

Target Variables (Reading Ease): The study measures three online eye-movement metrics known to index processing difficulty:
1. Total Fixation Duration (TF): Sum of fixation durations on a word.
2. Skip Rate (SR): Fraction of words skipped.
3. Regression Rate (RR): Number of backward saccades per word.
Evaluation Metric: The quality of a readability scoring method ( $M$ ) is evaluated by the Pearson correlation ( $r$ ) between the difference in scores ( $\Delta Score_M$ ) predicted by the method for the original vs. simplified text, and the difference in reading ease ( $\Delta ReadingEase$ ) observed in the eye-tracking data for the same text pairs.
$Eval_M = \text{Pearson\_corr}(\Delta Score_{M,T}, \Delta ReadingEase_{E,T})$

Systems Evaluated

The framework benchmarks 20 prominent readability scoring methods against theoretically motivated psycholinguistic measures:

Traditional Formulas: Flesch Reading Ease, Dale-Chall, Gunning Fog, ARI, Coleman-Liau, Flesch-Kincaid.
Modern NLP Methods: Supervised models (CML2RI, CAREC, CARES, Sentence-BERT) and unsupervised measures (Perplexity, Syntactic Log-Odds Ratio).
Commercial Systems: Lexile Text Analyzer and TextEvaluator (ETS).
Large Language Models (LLMs): Six frontier models (Llama 3.3, GPT-4o, GPT-5, Gemini 2.0/2.5, Claude Sonnet 4) prompted for grade-level or score-based annotations.
Psycholinguistic Baselines: Memory-related measures (Idea Density, Integration Cost, Embedding Depth) and information-theoretic measures (Surprisal, Entropy, Uniform Information Density, Pseudo Log Likelihood), alongside standard word properties (Word Length, Word Frequency).

Key Results

The evaluation reveals a significant gap between current ARA methods and the cognitive reality of reading ease.

Poor Performance of ARA Methods: Traditional formulas, modern NLP systems, commercial tools, and frontier LLMs all demonstrate low to non-significant correlations with real-time reading ease measures. Notably, there is no observed pattern of improvement in performance over time, despite the increasing sophistication of LLMs. Commercial systems (Lexile, TextEvaluator) showed particularly low predictivity.
Superiority of Psycholinguistic Word Properties: Standard word properties known to predict reading times in psycholinguistics outperformed all ARA methods. The "big three" predictors—Word Length, Word Frequency, and Surprisal—consistently yielded strong correlations.
Surprisal as the Best Predictor: Surprisal (the negative log probability of a word in context) emerged as the most predictive index of reading ease overall. It outperformed memory-based predictors (Idea Density, Integration Cost) and other information-theoretic measures.
Robustness: These findings hold across:
- Reader groups (L1 and L2 speakers).
- Reading regimes (comprehension vs. information seeking).
- Textual units (sentences and passages).
- Different eye-tracking measures (including First Fixation, Gaze Duration, and Reading Speed).
Content Control: The study demonstrates that the lack of content control in traditional evaluations (direct prediction of reading times) inflates the apparent performance of many ARA methods, as they often capture topical difficulty rather than linguistic complexity. The content-controlled approach reveals that methods like Surprisal, Entropy, and Word Length remain stable, while others degrade significantly when topic is controlled.

Significance and Claims

The paper claims to provide the first comprehensive picture of the limited ability of nearly a century of ARA methods to account for how humans experience texts in real time.

Limitation of Current Methods: The results highlight a critical limitation in widely used readability scoring methods, suggesting they are poor predictors of reading ease in adults compared to simple psycholinguistic word properties.
Caution in Application: The authors call for greater caution in using existing ARA methods and LLMs for readability scoring, particularly in high-stakes scenarios, as they may not accurately reflect the cognitive load of reading.
Need for New Approaches: The findings advocate for the development of new, cognitively driven readability scoring approaches that better account for real-time reading experiences.
Utility of Eye Tracking: The study underscores the utility of real-time behavioral benchmarks (eye tracking) for readability research, offering a continuous, interpretable alternative to offline comprehension tests and discrete human ratings.
Role of Surprisal: The paper suggests that Surprisal, computable via readily available language models, serves as a robust, simple, and general measure of readability that should be more broadly adopted.

The authors maintain modesty regarding the generalizability of their findings, noting that the study is restricted to English, adult participants, and the "intuitive" human simplification approach. They identify the need for future work to extend these evaluations to children, the elderly, other languages, and automated simplification contexts.