🧠 neuroscience

Neural tracking of surprisal and semantic distance in naturalistic movie viewing

Using fMRI data from participants watching a full-length film, this study demonstrates that the human brain tracks both surprisal and semantic distance as unique predictors of neural activity in bilateral temporal and frontal regions during naturalistic audiovisual language comprehension.

Original authors: O'Leary, R. M., Smith, H. C., Myers, E. B., Reilly, J., Peelle, J. E.

Published 2026-09-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: O'Leary, R. M., Smith, H. C., Myers, E. B., Reilly, J., Peelle, J. E.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

When we listen to a story, our brains are not just passive recorders of sound; they are active predictors, constantly guessing what comes next based on what we have already heard. This process relies on two distinct but related mental tools. The first is a sense of probability: how likely is the next word to appear given the words that came before it? If a sentence ends with "The sky is," our brain strongly expects the word "blue," and the arrival of that word feels unsurprising. If the sentence ends with "The sky is," and the next word is "banana," the brain is caught off guard. This feeling of being caught off guard is known as surprisal. The second tool is a measure of meaning, or how closely related the current word is to the ideas we have just processed. Even if a word is not the most statistically likely one, it might still fit perfectly with the theme of the conversation. Understanding how the brain uses these two different types of information—statistical likelihood and semantic relatedness—to make sense of language is a central question in neuroscience, especially when we are immersed in a rich, real-world environment like watching a movie.

A team of researchers set out to explore how the human brain tracks these two specific signals while people watch a full-length film. They recruited twenty adults who had never seen the 2009 movie 500 Days of Summer and asked them to watch the entire film, which lasts just over ninety minutes, while lying inside a functional magnetic resonance imaging scanner. This machine allowed the scientists to watch the brain's activity in real time as the story unfolded. To understand what the brain was doing, the researchers first broke down the movie's audio and visual tracks into thousands of tiny data points. They measured everything from the brightness of the screen and the movement of the camera to the loudness of the dialogue and the frequency of specific words. They then calculated two complex numbers for every word spoken in the film: one representing how surprising that word was based on the previous context, and another representing how semantically distant or close that word was to the words that came before it.

Using a sophisticated computer modeling technique, the researchers built a series of step-by-step predictions to see which of these data points could best explain the patterns of activity they saw in the brain. They started with the simplest factors, such as the visual changes on the screen and the raw volume of the sound. These basic features successfully predicted activity in the back of the brain, where visual processing happens, and in the sides of the brain, where sound is first received. As they added more complex language features, such as how often a word is used in general or how many similar-sounding words exist, the predictions improved, spreading to areas involved in language and memory. Finally, they added the two main variables of interest: the statistical surprise of the words and their semantic distance. The results showed that including these two factors significantly improved the ability to predict brain activity, particularly in the superior temporal gyrus, a region on the sides of the brain known for processing speech, and in the left frontal lobe.

The most critical discovery was that these two factors were not just doing the same job. Even after the researchers accounted for the influence of one, the other still explained unique changes in brain activity. This suggests that the brain is tracking both the statistical probability of what comes next and the conceptual relationship between ideas simultaneously, using two complementary systems. While both signals activated similar regions in the temporal lobes, the statistical surprise of words also showed a unique connection to a specific area in the cerebellum, a part of the brain at the base of the skull often associated with coordination and timing. This finding indicates that the brain's prediction mechanisms are nuanced, handling the "what is likely to happen" and "what fits the meaning" as separate but parallel streams of information.

These results are significant because they move beyond the controlled, quiet environments of traditional laboratory studies. Previous research on language prediction often relied on people listening to isolated sentences or reading text in silence. By using a full movie, the researchers demonstrated that the brain continues to track these subtle linguistic signals even when it is bombarded with complex visual scenes, music, and emotional storytelling. The study confirms that the brain's ability to anticipate language is robust and deeply integrated into how we experience the world, operating effectively even when the input is rich, multimodal, and unpredictable. The work does not claim to have solved the entire mystery of language processing, but it provides strong evidence that our brains are constantly running two different kinds of checks on the words we hear, ensuring we understand not just what is being said, but why it matters in the flow of the story.

Technical Summary: Neural Tracking of Surprisal and Semantic Distance in Naturalistic Movie Viewing

Problem Statement
Understanding natural language requires listeners to integrate incoming acoustic input with prior linguistic and thematic knowledge, a process heavily reliant on prediction. While surprisal (the negative logarithm of a word's probability given prior context) is a well-established metric for predictive difficulty, it is a composite measure blending word frequency, syntax, and semantics, making it difficult to isolate specific psycholinguistic factors. In contrast, semantic distance offers a more specific metric of conceptual relatedness within a high-dimensional semantic space.

Although both metrics are known to correlate with neural activation in language regions (particularly the superior temporal gyrus, STG), it remains unclear whether they reflect shared underlying neural representations or dissociable processes. Furthermore, existing research has largely relied on auditory-only speech, leaving a gap in understanding how these metrics operate within rich, naturalistic audiovisual contexts where visual scenes provide additional constraints. This study addresses whether surprisal and semantic distance make unique, dissociable contributions to neural responses during naturalistic movie viewing.

Methodology
The authors analyzed fMRI data from 20 native English-speaking adults (aged 19–53) who viewed the full-length film 500 Days of Summer (91.17 minutes) during continuous scanning. The study employed a series of hierarchical voxelwise encoding models to predict BOLD responses.

Feature Extraction:
- Low-level Controls: Visual features (luminance, contrast, color, motion, shot boundaries, face detection) were extracted using OpenCV. The auditory amplitude envelope was calculated as short-time RMS amplitude.
- Word-level Controls: Lexical frequency (SUBTLEX-US), phonological neighborhood density, and content word rate were included.
- Context-Sensitive Language Features:
  - Surprisal: Calculated using a pretrained GPT-2 model (117M parameters) based on the conditional probability of a word given the preceding 10 words of context.
  - Semantic Distance: Calculated using the SemanticDistance R package and GloVe vectors trained on the Corpus of Contemporary American English. Distance was defined as 1 minus cosine similarity between a word and a rolling 5-word window of prior content words.
Modeling Procedure:
- A hierarchical approach was used to assess incremental predictive power:
  - Model 1: Visual features only.
  - Model 2: Visual + Auditory envelope.
  - Model 3: Visual + Auditory + Word-level features.
  - Model 4: All previous features + Surprisal and Semantic Distance.
- Unique Contributions: To isolate unique variance, two intermediate models were constructed between Model 3 and Model 4: one adding only semantic distance, and one adding only surprisal.
- Validation: Models utilized 10-fold cross-validation with nested 5-fold inner loops for ridge regularization parameter selection. Predictors were expanded using a Finite Impulse Response (FIR) filter with lags of 0–8 TRs to capture hemodynamic delays without assuming a canonical response function.
- Inference: Group-level nonparametric inference (10,000 sign-flip permutations) with Threshold-Free Cluster Enhancement (TFCE) and family-wise error correction ( $p < .05$ ) was used to identify significant clusters.

Key Results

Hierarchical Progression: Model performance improved systematically as feature sets became more linguistically complex. Visual features predicted posterior occipital activity; adding the auditory envelope extended predictivity to bilateral STG; word-level features broadened prediction across temporal, frontal, and cerebellar regions.
Incremental Gains: The addition of context-sensitive features (surprisal and semantic distance) in Model 4 significantly improved prediction performance over Model 3, particularly in bilateral temporal cortex and the left inferior frontal gyrus (IFG).
Unique Contributions: Crucially, both metrics explained unique variance in neural activity after accounting for the other:
- Semantic Distance: Uniquely predicted activity in bilateral STG (anterior and posterior divisions), the left temporal pole, and the left parietal operculum.
- Surprisal: Uniquely predicted activity in bilateral STG, the left middle temporal gyrus, the left IFG, and notably, the right cerebellum (Crus I and II).
Spatial Overlap vs. Dissociation: While the spatial distributions of unique effects were largely overlapping (primarily in the STG), the statistical independence of the predictors confirmed that they capture non-redundant aspects of neural processing.

Significance and Claims
The paper claims that its primary contribution is demonstrating that surprisal and semantic distance are complementary but partially dissociable neural metrics even within a complex, multimodal (audiovisual) environment.

Beyond Auditory-Only: The findings extend previous work by showing that these linguistic predictors remain robust and meaningful when visual scene information is present, suggesting that the brain tracks these probabilistic and semantic relationships regardless of the modality richness.
Distinct Mechanisms: The fact that both metrics explain unique variance in the bilateral STG suggests that the brain utilizes both broad probabilistic prediction (surprisal) and specific semantic integration (semantic distance) simultaneously. The authors interpret the STG effects as reflecting top-down feedback where context facilitates the processing of word-form details.
Cerebellar Involvement: The identification of the right cerebellum (Crus I/II) as a region uniquely sensitive to surprisal supports emerging literature linking the cerebellum to linguistic prediction and error processing.
Methodological Rigor: By using hierarchical encoding models with strict controls for low-level features and word-level statistics, the study provides a stricter test of independent contributions than prior work that often used unimodal stimuli or less context-sensitive models.

The authors remain modest regarding generalizability, noting the study was limited to a single movie genre and that the high correlation between surprisal and semantic distance in naturalistic speech prevents experimental manipulation of these factors independently. They conclude that while semantic knowledge is distributed across a wide network, the immediate consequences for speech comprehension are expressed most strongly as modulations within the STG.

Technical Summary: Neural Tracking of Surprisal and Semantic Distance in Naturalistic Movie Viewing

More like this