💬 NLP

On the Proper Treatment of Units in Surprisal Theory

This paper proposes a unified framework to disentangle the definition of linguistic units from evaluation regions in surprisal theory, arguing that tokenization should be treated as an implementation detail rather than a scientific primitive to ensure explicit and rigorous modeling of human processing effort.

Original authors: Samuel Kiegeland, Vésteinn Snæbjarnarson, Tim Vieira, Ryan Cotterell

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Samuel Kiegeland, Vésteinn Snæbjarnarson, Tim Vieira, Ryan Cotterell

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Wrong Ruler"

Imagine you are a psychologist trying to measure how hard it is for a human brain to read a sentence. You have a theory called Surprisal Theory, which says: The harder a word is to predict, the more mental effort it takes to read.

To test this, you need a "ruler" to measure that effort. In the past, researchers used a ruler that matched the words perfectly (like measuring a table in inches). But today, we use powerful AI models (like GPT-2) to do the measuring. The problem? These AI models don't speak "human words." They speak in "tokens."

Think of a token like a Lego brick.

A human sees the word "don't".
The AI sees two bricks: "don" and "'t".
A human sees the word "words".
The AI sees one brick: "words".
A human sees the word "equal".
The AI sees one brick: "equal".

Because the AI's "bricks" (tokens) don't line up with our "words," researchers have been trying to glue the AI's bricks together to match human words. They've been using "ad hoc" (make-shift) glue. Sometimes they glue a space to the front of a word; sometimes to the back. Sometimes they ignore punctuation.

The Paper's Main Claim: This "gluing" is messy and inconsistent. It's like trying to measure a room with a ruler that changes its own length every time you look at it. The authors argue that the choice of what counts as a "unit" (a word, a letter, a phrase) is a scientific decision, not just a technical detail. We need to stop letting the AI's internal brick structure dictate our science.

The Solution: "Bring Your Own Units"

The authors propose a new framework where the researcher decides exactly what a "unit" is before looking at the AI.

You choose the unit: Do you want to study whole words? Individual letters? Or maybe specific parts of speech?
You translate the AI: Instead of forcing the AI to change, you build a "translator" (a mathematical tool called a transducer) that converts the AI's weird bricks into your chosen units.

The Analogy:
Imagine the AI is a chef who only speaks in "grams of flour, sugar, and eggs." You, the scientist, want to know how much "cake" is being made.

Old Way: You try to guess how many grams make a cake based on the chef's random measurements.
New Way: You build a machine that takes the chef's grams and automatically assembles them into perfect "cakes" (or "cupcakes," or "muffins") based on your recipe. The chef doesn't change; your machine just interprets the output correctly for your specific question.

The Experiment: Testing Different Rulers

To prove their point, the authors ran an experiment using the MECO dataset (a collection of eye-tracking data from people reading text). They tested four different "rulers" (unit inventories) to see which one best predicted how long people's eyes stayed on a word:

Tokens: Using the AI's native bricks (e.g., "don" and "'t" as separate things).
Characters: Measuring every single letter (e.g., d, o, n, ', t).
Acontextual Words: Splitting text by simple rules (e.g., "cut at every space"). This is like cutting a string of beads at every gap, regardless of what the beads look like.
Contextual Words: Using smart rules (like the Penn Treebank guidelines) that understand context. For example, knowing that a comma in "1,000" is part of the number, but a comma in "Hello, world" is a pause.

The Results: It Matters What You Measure

The study found that changing the ruler changes the results.

Whole Words (Contextual): When they used "smart" word units that respected punctuation and grammar, the AI's predictions matched human eye movements very well.
Tokens: The AI's native bricks worked okay, but not as well as the smart words.
Characters: Measuring letter-by-letter didn't work well for predicting reading time. Why? Because humans rarely stop their eyes on a single letter; they look at whole words.
The "Glue" Problem: They found that how you handle spaces (leading vs. trailing) changed the math significantly.

The Takeaway:
If you use a bad ruler, you might conclude that "Surprisal Theory is wrong" or "AI models are bad," when in reality, you just measured the wrong thing.

The Core Lesson

The paper concludes that tokenization is just a technical detail, like the brand of the camera lens. It shouldn't be the star of the show.

Old Mindset: "The AI uses tokens, so we must study tokens."
New Mindset: "I want to study how humans process words. I will take the AI's output and mathematically convert it into words so I can answer my scientific question correctly."

By treating the definition of a "unit" as a deliberate scientific choice rather than a default setting, researchers can get clearer, more accurate answers about how the human brain processes language.

1. Problem Statement

Surprisal theory posits that human cognitive processing effort is determined by the negative log-probability (surprisal) of an upcoming linguistic unit given its context. However, a critical methodological gap exists in current empirical research:

The Mismatch: Experimental stimuli are typically segmented into linguistically motivated units (e.g., words, morphemes), but modern pretrained language models (LMs) operate on fixed token alphabets (e.g., Byte-Pair Encoding) that do not align with these linguistic units.
The Consequence: Researchers currently rely on ad hoc, post-hoc heuristics to map model tokens to linguistic units. These procedures often conflate two distinct modeling choices:
1. Unit Inventory ( $U$ ): The definition of the linguistic unit (e.g., word, character, phrase).
2. Regions of Interest (ROIs): The specific spans over which predictions are evaluated (e.g., aggregating word-level surprisals into sentence-level metrics).
The Flaw in Existing Approaches: Recent attempts to formalize this mapping (e.g., Oh & Schuler, 2024; Pimentel & Meister, 2024) assume that unit boundaries can be determined by a fixed partition of the symbol alphabet (e.g., whitespace always marks a boundary). The authors argue this is linguistically inadequate (e.g., a comma is a boundary in "end, he" but internal in "1,000") and mathematically inconsistent when dealing with context-dependent tokenization.

2. Methodology

The authors propose a unified formal framework to disentangle unit definitions from tokenization, treating tokenization as an implementation detail rather than a scientific primitive.

A. Formal Framework

Separation of Concerns: The paper distinguishes between the Symbol Alphabet ( $\Sigma$ ) (the LM's native tokens/characters) and the Unit Inventory ( $U$ ) (the researcher's chosen linguistic units).
Unit Parser ( $\rho$ ): A stochastic map $\rho: \Sigma^* \to U^*$ that converts symbol strings into unit strings. The authors assume $\rho$ is deterministic for practical purposes.
Pushforward Probability: To derive a probability distribution over units ( $p_U$ ) from a symbol-level model ( $p_\Sigma$ ), they use the pushforward operation:
$p_U(u) = \sum_{\sigma \in \rho^{-1}(u)} p_\Sigma(\sigma)$
This sums the probabilities of all symbol strings that map to a specific unit $u$ .

B. Transduced Language Models

To handle the computational challenge of summing over potentially infinite symbol strings, the authors introduce Finite State Transducers (FSTs):

Regular Unit Inventories: They assume $U$ is a "regular" subset of strings over a finite alphabet $\Xi$ .
The Transducer ( $f$ ): A finite transducer $f = h \circ \rho$ maps symbol strings $\Sigma^*$ to strings over a finite alphabet $\Delta = \Xi \cup \{SEP\}$ , where $SEP$ is a separator symbol.
Algorithm: By composing the LM with the transducer, they can compute the pushforward distribution efficiently using beam search approximations (Snæbjarnarson et al., 2026). This allows the calculation of surprisal for arbitrary unit inventories without retraining the LM.

C. Experimental Design

Dataset: The Multilingual Eye-movement Corpus (MECO), specifically the English portion (46 readers, 12 Wikipedia excerpts).
Unit Inventories Tested:
1. Tokens: Native GPT-2 tokens.
2. Characters: Individual characters.
3. Acontextual Words: Words split by fixed delimiters (whitespace), testing both leading and trailing whitespace attribution.
4. Contextual Words: Words split using Penn Treebank (PTB) rules (e.g., handling punctuation like "1,000" vs. "end, he").
Analysis:
- Baseline: Log-normal Generalized Additive Mixed Models (GAMMs) predicting reading times using unit length, unigram surprisal, and spillover effects.
- Target: The baseline plus contextual surprisal.
- Metric: Improvement in held-out log-likelihood ( $\Delta llh$ ) via leave-one-out cross-validation.

3. Key Contributions

Theoretical Disentanglement: The paper formally separates the choice of units from the choice of tokenization, arguing that unit selection should be a primary modeling decision driven by the scientific question, not a byproduct of the LM's architecture.
Generalized Formalism: They provide a "bring your own units" framework using transducers, allowing researchers to define arbitrary, context-sensitive unit inventories (e.g., PTB-style tokenization) and compute surprisal directly from pretrained LMs.
Critique of Current Heuristics: They demonstrate that fixed-partition approaches (treating whitespace as a universal boundary marker) fail to capture linguistic reality (e.g., punctuation attachment) and introduce mathematical inconsistencies regarding unit identity.
Practical Toolkit: The authors release code and transducers to convert token-level models into character-level or context-aware word-level models.

4. Results

The authors evaluated the predictive power of surprisal across the four unit inventories on three reading-time measures: First Fixation (FF), Gaze Duration (GD), and Total Reading Time (TRT).

Word-like Inventories: Surprisal significantly improved the prediction of reading times for Gaze Duration and Total Reading Time across all word-like inventories (Tokens, Acontextual, Contextual).
First Fixation: Surprisal was significant for Tokens, Contextual Words, and Acontextual (trailing), but not for Acontextual (leading). This suggests that leading-whitespace attribution (common in BPE) may misalign with early cognitive processing.
Character Level: Character-level surprisal showed no significant improvement in predicting reading times. The authors attribute this to the fact that single characters are rarely fixated individually; the cognitive unit of processing is larger than a single character.
Contextual vs. Acontextual: Contextual word inventories (handling punctuation correctly) generally performed as well as or slightly better than acontextual ones, particularly for complex punctuation cases, validating the need for context-sensitive segmentation.
Non-Comparability of Absolute Scores: The study highlights that absolute log-likelihoods and $\Delta llh$ values cannot be directly compared across different unit inventories because the inventories induce different observation granularities and baseline predictors (e.g., unit length).

5. Significance

Methodological Rigor: The paper establishes that treating tokenization as a fixed constraint limits the validity of psycholinguistic conclusions. Researchers must explicitly define their unit inventory.
Reproducibility and Flexibility: By providing a transducer-based framework, the authors enable the community to test hypotheses about specific linguistic units (morphemes, phrases, discourse units) without the prohibitive cost of retraining LMs from scratch.
Linguistic Adequacy: The results support the view that context-sensitive segmentation (like PTB) is superior to simple delimiter-based splitting for modeling human reading behavior, particularly regarding punctuation.
Future Directions: The framework is extensible to other dependent variables (neural signals, lossy-context surprisal) and other languages, provided appropriate transducers can be constructed.

In summary, the paper argues that tokenization is an implementation detail, while unit selection is a scientific choice. By decoupling the two via transducers, researchers can ensure their surprisal metrics accurately reflect the linguistic units they intend to study.