💬 NLP

Energy-Based Transformers as Predictors of Reading Difficulty

This paper introduces energy-based transformers as a novel, unified predictor of human reading difficulty that outperforms and subsumes existing measures like surprisal and attention entropy across multiple reading-time corpora and syntactic structures.

Original authors: Jakub Dotlacil, Ece Takmaz

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jakub Dotlacil, Ece Takmaz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: A New Way to Measure "Reading Trouble"

Imagine you are reading a book. Sometimes, a sentence flows smoothly, and your eyes glide over the words. Other times, you hit a bump, your eyes stop, and you have to re-read a phrase to figure out what it means. In the world of computers, we call this "reading difficulty."

For a long time, researchers have used two main tools to predict when a human reader will hit a bump:

Surprisal: This measures how unexpected a word is. If you say, "The cat sat on the...", the word "mat" is expected (low surprisal). If you say "The cat sat on the... toaster," that's a shock (high surprisal).
Attention Entropy: This measures how "confused" the computer's focus is. Imagine your attention is a flashlight. If the flashlight is focused tightly on one word, entropy is low. If the flashlight is wobbling and shining on ten different words at once, entropy is high (and that usually means the sentence is hard to process).

The Problem: Usually, researchers need both tools to get a good prediction. Sometimes "Surprisal" explains the difficulty, and sometimes "Attention Entropy" does. It's like trying to fix a car by checking the engine and the tires separately every time.

The New Solution: This paper introduces a third tool called Energy. The authors tested a special type of AI model called NRGPT (Energy-based GPT). They found that this "Energy" measure is like a universal translator. It seems to combine the best parts of both "Surprisal" and "Attention Entropy" into a single number.

How the "Energy" Model Works

To understand "Energy," imagine the AI model isn't just predicting the next word; it's trying to settle down into a comfortable position.

The Landscape Analogy: Think of the AI's understanding of a sentence as a hilly landscape.
- Low Energy (The Valley): The sentence makes perfect sense. The words fit together easily, like a puzzle piece snapping into place. The AI is "relaxed."
- High Energy (The Mountain Peak): The sentence is confusing or awkward. The AI is "straining" to make sense of it. It's like trying to balance a ball on top of a steep hill.

In this model, the AI doesn't just guess the next word; it physically (mathematically) rolls down this hill to find the most stable, "low-energy" way to understand the sentence. The paper argues that the amount of effort (energy) it takes to roll down that hill is a perfect predictor of how hard a human has to work to read that sentence.

What They Tested

The researchers ran two main experiments to see if this "Energy" idea actually works.

1. The "Relative Clause" Test (The Tricky Sentence)

They looked at a classic puzzle in language: the difference between two types of sentences.

Sentence A (Easy): "The firemen that called the residents attacked the house." (Subject Relative)
Sentence B (Hard): "The firemen that the residents called attacked the house." (Object Relative)

Humans find Sentence B much harder to read at the verb ("called").

Old Tools: "Surprisal" failed to see the difference here. "Attention Entropy" saw the difference but missed other parts of the sentence.
The New Tool: The Energy measure caught the difficulty in Sentence B perfectly. It was high when the sentence was hard and low when it was easy. It successfully captured the "bump" that humans feel.

2. The "Real Book" Test (Reading Speed)

They fed the model thousands of real sentences from books and compared its "Energy" scores against actual data from people reading those books (measuring how long their eyes stayed on each word).

The Result: The "Energy" score was a very strong predictor of reading time. In fact, it was so good that even when they added "Surprisal" to the mix, the "Energy" score still added new, useful information. It wasn't just repeating what the other tools said; it was finding trouble spots the others missed.

Why This Matters

The authors suggest that "Energy" might be the Swiss Army Knife of reading research.

Instead of needing a separate tool for "unexpected words" and another for "confusing focus," we might just need this one "Energy" number.
It connects the way computers process language to how human memory works (specifically, how our brains store and retrieve associations).

Important Limits (What They Didn't Say)

The paper is careful to say what they didn't prove:

They only tested one specific version of this AI model. We don't know if it works for every other AI model yet.
They only tested it on reading speed. They didn't test if it predicts brain scans (fMRI) or other types of thinking tasks.
They didn't test it on different languages or complex medical diagnoses.

In short: The paper shows that a new way of measuring AI "effort" (Energy) is a powerful, single tool that can predict when humans will struggle to read a sentence, potentially replacing the need for multiple, separate measurement tools.

Problem

Transformer language models have become established tools for modeling human sentence processing, primarily through two measures: surprisal (log inverse probability) and attention entropy. While surprisal effectively predicts processing load based on lexical predictability, and attention entropy captures aspects of memory interference, neither measure alone provides a comprehensive fit for reading-time data. Current research often requires combining these complementary measures to model human processing accurately. Furthermore, while the conceptual link between transformer attention and human memory is intuitive, there is a lack of a principled formal link connecting transformers to the broader literature on associative memory and Hopfield networks. The authors identify a need for a single, unified measure that can capture both predictability-driven and integration-driven processing difficulties, potentially bridging the gap between transformer architectures and cognitive theories of associative memory.

Methodology

The authors investigate Energy-Based Transformers, specifically the NRGPT (Energy GPT) architecture, as a predictor of reading difficulty. NRGPT reformulates the transformer forward pass as gradient descent on a scalar energy function defined over token states.

Model Architecture: NRGPT operates similarly to a standard GPT but employs a parallel structure (attention and feedforward layers apply in parallel) and recurrence (the transformer block is applied iteratively $k$ times). Each iteration represents a descent step on an energy landscape.
Energy Function: The model associates a scalar energy $E_A$ $E_{A}$ with each token position $A$ $A$ , decomposed into attention energy ( $E^{AT}_A$ $E_{A}^{A T}$ ) and feedforward energy ( $E^{FF}_A$ $E_{A}^{F F}$ ).
- $E^{AT}_A$ is derived from attention scores and relates to attention entropy.
- $E^{FF}_A$ is based on dense associative memory.
- Intuitively, high energy indicates difficulty in integrating a token into the context, while low energy indicates ease of integration.
Experimental Design:
1. Experiment 1 (Controlled): The authors analyzed 24 pairs of Subject Relative Clauses (SRC) and Object Relative Clauses (ORC). They compared NRGPT energy (at layers 5 and 6), surprisal, and attention entropy against known human processing asymmetries (specifically the difficulty of the embedded verb in ORCs vs. SRCs and the noun phrase onset).
2. Experiment 2 (Corpora): The authors fitted linear mixed-effects models to three reading-time corpora: Natural Stories (self-paced reading), UCL eye-tracking, and UCL self-paced reading. They compared the predictive power of NRGPT energy (layers 5 and 6) and surprisal (from NRGPT and GPT-2-small) against reading times.

Key Contributions

First Application in Psycholinguistics: This is the first exploration of an energy-based transformer measure in computational psycholinguistics for fitting human sentence processing data.
Theoretical Link: The work establishes a principled formal link between transformer processing and dense associative memory (modern Hopfield networks), moving beyond conceptual analogies to a mathematical framework where processing is viewed as energy minimization.
Unified Predictor: The authors demonstrate that the scalar energy function can serve as a single unified predictor that subsumes effects previously requiring both surprisal and attention entropy.

Results

Relative Clause Processing:
- Surprisal failed to differentiate between SRC and ORC verbs (consistent with GPT-2 findings) but predicted difficulty at the noun phrase onset.
- Attention Entropy predicted the difficulty of the embedded verb in ORCs but not the noun phrase onset.
- NRGPT Energy (Layer 5) successfully predicted the difficulty contrast for both the embedded verb and the noun phrase onset.
- Decomposition analysis revealed that the energy effect at Layer 5 was driven primarily by the expected attention score (a component of attention energy) and feedforward energy, rather than attention entropy alone.
Reading-Time Corpora:
- Across all three corpora, the NRGPT energy measure was a robust predictor of reading times.
- Energy provided a significant fit beyond surprisal in all three datasets.
- In the Natural Stories corpus, surprisal provided a better fit than energy alone, but energy remained significant when combined with surprisal.
- In the UCL corpora (both eye-tracking and self-paced), energy provided a better fit than surprisal.
- Layer Specificity: Energy at the fifth layer was particularly predictive for self-paced reading data, while energy at the sixth layer showed strong predictive power for eye-tracking data (though the authors note the eye-tracking dataset had fewer observations).
- The correlation between Layer 5 energy and surprisal was moderate ( $r=0.41$ ), suggesting Layer 5 energy captures surprisal-like effects but also distinct integration-driven processing costs.

Significance

The paper claims that the energy measure offers a single, straightforward link between transformers and processing difficulty, potentially replacing the need for multiple complementary measures. By framing token integration as a descent on an energy landscape, the work connects computational psycholinguistics directly to the literature on associative memory. The authors suggest that energy-based transformers may capture integration-driven difficulty that surprisal misses, while still tracking predictability. This approach provides a new tool for investigating the connection between memory and processing in neural language models, grounded in the theoretical framework of dense associative memory rather than just conceptual intuition. The authors remain modest, noting that this is a first exploration and that generalizability to other architectures, model sizes, and languages requires further investigation.