💬 NLP

Assessing Cognitive Effort in L2 Idiomatic Processing: An Eye-Tracking Dataset

This paper introduces and validates an eye-tracking dataset of Portuguese L1 speakers of English across all proficiency levels, demonstrating that even entry-level 60 Hz hardware can effectively capture the cognitive costs of L2 idiomatic processing and provide a benchmark for evaluating both human and artificial language models.

Original authors: Eduardo Santos, Juliana Carvalho, César Rennó-Costa

Published 2026-05-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Eduardo Santos, Juliana Carvalho, César Rennó-Costa

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Reading Idioms is Like Solving a Puzzle

Imagine you are reading a sentence like, "The chef spilled the beans."

If you are a native English speaker: Your brain instantly sees a picture of a messy kitchen. You don't think about the actual beans; you just know the phrase means "someone revealed a secret." It's like recognizing a friend's face instantly without analyzing their nose or eyes.
If you are learning English (a second-language learner): Your brain might first see a picture of a clumsy chef dropping actual beans on the floor. It tries to make sense of the words literally. Then, suddenly, it realizes, "Wait, that doesn't make sense in this story!" Your brain has to hit the "rewind" button, go back, and figure out the secret meaning. This "rewind" takes extra mental energy.

This paper is about measuring exactly how much that "rewinding" costs the brain.

The Experiment: The "Gaze Tracker"

The researchers wanted to see this mental "rewind" happen in real-time. They couldn't just ask people, "Was that hard?" because people aren't always aware of their own thinking. Instead, they used eye-tracking.

Think of the eye-tracker as a high-speed camera that watches where your eyes land and how long they stay there.

Fixations: When your eyes stop to read a word (like a car stopping at a red light).
Regressions: When your eyes jump backward to re-read a word (like a car backing up to check a turn).

The researchers found that when learners struggle with an idiom, their eyes jump backward (regress) much more often than when they read normal sentences.

The Participants: The "Language Ladder"

The study didn't just look at random people. They recruited students from Brazil who speak Portuguese as their first language and are learning English. They sorted these students into six levels, from A1 (Total Beginner) to C2 (Master/Fluent).

Think of this like a video game with levels:

Level A1/A2 (Beginners): They are still learning the rules. When they hit an idiom, they get stuck, back up, and try to decode it word-by-word.
Level C1/C2 (Experts): They are playing on "autopilot." They recognize the idiom immediately, just like native speakers, and their eyes don't need to jump backward.

The Hardware: The "Entry-Level Camera"

One interesting part of the paper is that they didn't use a super-expensive, high-tech lab camera. They used a standard, affordable eye-tracker (the Tobii Pro Spark) that takes 60 pictures per second.

The authors argue that even though this is "entry-level" gear, it's like having a good enough camera to see a car stop and back up. You don't need a camera that takes 1,000 pictures per second to see that the driver made a mistake; 60 pictures per second is enough to catch the "rewind" moment.

The Results: The "Rewind" Count

The data confirmed their theory. They counted how many times each person's eyes jumped backward (regressions) while reading.

Beginners (B1): They had a huge number of "rewinds." Their eyes were jumping back and forth constantly, showing they were working very hard to understand the meaning.
Experts (C2): They had very few "rewinds." Their eyes moved smoothly forward, showing they understood the idiom instantly.

There was a clear pattern: The better the English, the fewer times the eyes had to go back.

Why This Matters (According to the Paper)

The researchers created a massive collection of this data (a "dataset") and made it free for other scientists and computer programmers to use.

They are doing this to help build better Artificial Intelligence (AI). Currently, AI models (like the ones that write essays or chat with you) are great at grammar but sometimes struggle with idioms. By feeding this eye-tracking data into AI, the researchers hope to teach the computer to understand how humans struggle with these phrases, so the AI can learn to process them more like a human does.

Summary

In short, this paper says:

Learning idioms is hard for non-native speakers because their brains try to interpret them literally first.
We can measure this struggle by watching their eyes jump backward.
We built a library of this eye-tracking data for learners at all skill levels.
This data proves that as you get better at a language, your brain stops "rewinding" and starts reading smoothly.
This data is now available to help computers learn to understand human language better.

Technical Summary: Assessing Cognitive Effort in L2 Idiomatic Processing

Problem Statement
The processing of Multi-Word Expressions (MWEs), particularly idioms, presents a persistent challenge in both cognitive science and computational linguistics. While native speakers often access idiomatic meanings via direct retrieval, Second Language (L2) learners frequently rely on a "literal-first" approach, incurring measurable cognitive costs. Current Natural Language Processing (NLP) architectures, including Large Language Models (LLMs), struggle with the nuances of figurative language and often function as "black boxes." Existing resources for studying this phenomenon are often limited by small sample sizes, homogeneous participant groups, or a lack of fine-grained temporal data linking eye movements to specific proficiency levels. There is a critical need for resources that capture the real-time cognitive effort of L2 learners to bridge human processing models with artificial systems.

Methodology
This study introduces a specialized eye-tracking dataset designed to investigate the processing of "Potentially Idiomatic Expressions" (PIEs) by Portuguese L1 speakers of English across all CEFR proficiency levels (A1–C2).

Experimental Design: The study utilized a matched-pairs design where participants read sentences containing target MWEs (e.g., "spill the beans") in two contexts: figurative and literal. The stimuli were primarily drawn from the MAGPIE corpus, ensuring that word strings could function as either idioms or literal phrases depending on context.
Participants: The dataset includes university students from the Federal University of Rio Grande do Norte (UFRN) and the Digital Metropolis Institute (IMD), stratified by self-reported and validated CEFR levels (A1–C2).
Data Acquisition: Data was collected using a Tobii Pro Spark eye tracker operating at 60 Hz. Although categorized as entry-level hardware, the study argues this sampling rate provides sufficient density (approx. 16.67 ms resolution) to detect macro-cognitive events like fixations and regressions. The environment was controlled for lighting and viewing distance, with a 9-point calibration and practice phase preceding the main task.
Data Processing: A custom, open-source pipeline was developed to transform raw gaze coordinates into psycholinguistic metrics. Since the hardware lacks onboard event detection, the pipeline used the PyGaze library to identify fixations via an Identification by Dispersion Threshold (I-DT) algorithm. Crucially, a custom heuristic was implemented to detect regressions (backward eye movements) by analyzing the sequential logic of fixation coordinates ( $X_n < X_{n-1}$ ) rather than relying solely on velocity thresholds, thereby mitigating signal-to-noise limitations of 60 Hz data.

Key Contributions

Stratified Eye-Tracking Dataset: The paper presents a structured dataset containing raw logs, processed files, and aggregated metrics for L2 learners across the full CEFR spectrum (A1–C2). This allows for the systematic investigation of how idiomatic processing strategies evolve during language acquisition.
Transparent Processing Pipeline: The authors provide a reproducible, open-source pipeline that bridges proprietary hardware formats and standard analysis libraries. This includes specific algorithms for detecting regressions in lower-frequency eye-tracking data, addressing a gap in existing tooling.
Validation of the "Re-analysis Hypothesis": The dataset serves as a cognitively grounded benchmark for the "Modeling Idiomaticity in Human and Artificial Language Processing" (MIA) initiative, aiming to align AI models with human-like figurative understanding.

Results
Preliminary analysis validates the dataset by demonstrating a strong inverse correlation between language proficiency and regressive eye movements:

B1 (Intermediate) Learners: Exhibited high heterogeneity and the highest number of regressions (Mean: 1,776; Max: 4,046), indicating acute cognitive demand and a recursive processing loop where literal decoding fails, necessitating re-analysis.
B2 to C1 Learners: Showed a progressive decrease in regression counts and variance, signaling a transition from effortful decoding to more stable processing strategies.
C2 (Proficient) Learners: Demonstrated low regression counts (Mean: 987) and minimal variance, clustering in the lower-left quadrant of fixation/regression distributions. This behavior mirrors native-like stability and direct lexical access.
Correlation: A robust positive correlation was found between total fixations and total regressions, confirming that regressions serve as a reliable index of cognitive workload rather than stochastic noise.

Significance
The paper positions this dataset as a foundational resource for the MIA initiative and the SemEval-2025 Task 1.1. Its significance lies in providing a "ground-truth" signal of cognitive friction observed in human learners, which can be used to train and evaluate AI models on non-compositional shifts. By capturing the millisecond-by-millisecond evolution of reading strategies from A1 to C2, the resource enables researchers to move beyond offline comprehension measures (like questionnaires) to analyze the dynamic cognitive processes underlying L2 idiomaticity. The authors note that while the dataset is currently limited to Brazilian Portuguese speakers and 60 Hz hardware, it establishes a scalable framework for future expansion and more fine-grained analysis of cognitive effort in language processing.