🧠 neuroscience

Adaptive sequential eye-movement sampling and replay under different task demands

This study demonstrates that human eye movements actively organize working memory by spontaneously aligning with latent structural patterns during encoding and shifting to a flexible, replay-like mechanism during maintenance when cognitive demands increase.

Original authors: Huang, Q., Doeller, C. F.

Published 2026-02-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Huang, Q., Doeller, C. F.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The Big Idea: Your Eyes Are Like a Librarian's Cart

Imagine your brain is a busy library, but it has a very small shelf. It can only hold a few books at a time (this is called working memory). If you try to shove too many books onto that tiny shelf at once, they get messy, and you might lose them.

This study asks a simple question: How does your brain keep these "books" organized when it's crowded?

The researchers discovered that your eyes do more than just look at things; they act like a smart librarian's cart. When you are trying to remember things, your eyes don't just wander randomly. They follow a specific, efficient path to organize the information, almost like a mental "replay" tape.

The Experiment: The Color and Pattern Game

The researchers set up a game to test this. Imagine you are shown three glowing shapes on a screen at the same time.

Each shape has a color (ranging from green to blue).
Each shape has a pattern (some are thick lines, some are thin lines).

Here is the secret trick: The researchers made a rule.

Thick lines always go with Green.
Thin lines always go with Blue.
The middle ones are in between.

So, even though the shapes are in different spots on the screen, your brain can secretly rank them: Greenest/Thickest (1) → Middle (2) → Bluest/Thinnest (3).

The participants had to memorize where these shapes were, and then answer questions about them later. While they did this, the researchers tracked their eye movements with high-speed cameras.

What They Found: The "Smart" vs. "Struggling" Strategies

The study revealed two different ways people used their eyes to help their memory, depending on how good they were at the task and how hard the task was.

1. The "Pro" Strategy (Encoding)

The Analogy: Imagine a chef who knows exactly where every ingredient is. When they start cooking, they grab the items in the perfect order: First the knife, then the onion, then the pan. They don't need to look around later; they organized it perfectly the first time.

What happened: The people who were very good at the memory task (the "Pros") used their eyes to scan the shapes in a perfect order (1 → 2 → 3) while the shapes were still on the screen.
The Result: Because they organized the information so well while looking at it, they didn't need to do much mental work later. Their memory was strong and stable.

2. The "Replay" Strategy (Maintenance)

The Analogy: Imagine a student who didn't organize their notes while studying. When the test starts, they panic. They have to close their eyes and mentally "re-read" their notes in order to remember what they wrote. They are replaying the information in their head.

What happened: The people who found the task harder (the "Strugglers") didn't organize the shapes well while looking at them. But, once the shapes disappeared, their eyes started moving in that same perfect order (1 → 2 → 3) while the shapes were gone.
The Result: Their eyes were "replaying" the sequence to help their brain hold onto the information. It was a backup plan to save their memory.

The Twist: When the Task Gets Harder

The researchers then made the game harder by showing four shapes instead of three. This is like trying to carry four heavy boxes instead of three.

The "Pros" got overwhelmed: Even the smart participants couldn't organize the four items perfectly while looking at them. The "encoding" strategy broke down.
The Shift: Suddenly, everyone started using the "Replay" strategy. Even the experts had to wait until the shapes disappeared and then use their eyes to mentally "replay" the order to keep the information safe.

Why This Matters

This study proves that eye movements are not just a side effect of thinking; they are a tool for thinking.

When things are easy: Your eyes help you organize information as you see it.
When things are hard: Your eyes help you re-organize and stabilize information after you've seen it, acting like a mental "save button."

It's like realizing that when you are in a rush, you don't just walk faster; you change your whole route to make sure you don't drop anything. Your eyes are constantly adapting to help your brain hold onto the world, even when the load gets too heavy.

In a Nutshell

Your eyes are the physical manifestation of your brain's "mental filing system." When you are calm, you file things away neatly as you look at them. When you are stressed or overwhelmed, your eyes start "replaying" the files in your head to make sure nothing gets lost.

1. Problem Statement

Human working memory (WM) is capacity-limited, necessitating efficient coding strategies to organize information. While eye movements are known to reflect memory accessibility, it remains unclear whether they play an active, organizational role in structuring mnemonic representations. Specifically, the study addresses two critical gaps:

Encoding: How are sequential sampling orders established when multiple items must be encoded simultaneously? Does the oculomotor system actively impose structure on concurrent inputs?
Maintenance: Does this sequential structure persist or re-emerge during the maintenance phase (when stimuli are invisible), acting as a behavioral analogue to neural "replay"?
Adaptivity: How does cognitive load modulate the reliance on encoding-based organization versus maintenance-based replay?

2. Methodology

Experimental Design

The authors conducted two experiments using a working memory paradigm with embedded latent structures:

Stimuli: Participants viewed multiple gratings (3 in Exp 1, 4 in Exp 2) presented simultaneously at fixed spatial locations. Each grating varied in spatial frequency and color (green-blue continuum).
Latent Structure: A systematic coupling was introduced: lower spatial frequencies were consistently paired with "greener" hues, and higher frequencies with "bluer" hues. This created an implicit ordinal structure (Item 1: lowest freq/greenest $\to$ Item N: highest freq/bluest).
Task:
1. Encoding: Participants viewed stimuli for 4s (Exp 1) or 5s (Exp 2) while free to move their eyes (except during central fixation cues).
2. Maintenance: Stimuli disappeared; participants maintained the items for 2s (Exp 1) or 2.5s (Exp 2).
3. Retrieval: A retro-cue directed attention to a specific location, followed by a probe (color or frequency) to test memory.
Participants: 89 total (46 in Exp 1, 40 in Exp 2).

Data Acquisition & Preprocessing

Eye Tracking: SR Research EyeLink 1000 Plus at 1,000 Hz.
Preprocessing: Blink detection and interpolation, exclusion of trials with excessive deviation from fixation.
Analysis Frameworks:
1. Pattern-Similarity Analysis: Quantified gaze-to-item distance time courses. Compared correlations between trials sharing the same spatial/feature configuration (within-pattern) vs. different configurations (between-pattern) to detect value-guided sampling.
2. Sequence Analysis (TDLM): Applied Temporally Delayed Linear Modeling (Liu et al., 2021) to cross-correlation matrices of gaze distances. Tested for specific transition models (e.g., $1 \to 2 \to 3$ ) against empirical data at various time lags.
3. Saccade Analysis: Analyzed first and second saccades following the retro-cue to verify if recall followed the learned ordinal sequence.

3. Key Contributions

Active Organization of Memory: Demonstrated that eye movements do not merely reflect memory but actively structure it. Participants spontaneously adopted a sequential sampling strategy ( $1 \to 2 \to 3$ ) based on the latent feature gradient during encoding.
Behavioral Replay: Provided evidence of covert replay during the maintenance phase. When encoding resources were strained (low performers or high load), participants exhibited sequential "re-visitation" of items in the correct order despite the absence of visual input.
Adaptive Mechanism: Revealed a dynamic shift in strategy based on cognitive demand and individual ability:
- High Performers (Low Load): Relied on robust sequential sampling during encoding.
- Low Performers (Low Load): Failed to organize during encoding but compensated with sequential replay during maintenance.
- High Load (4 items): Encoding-based organization was disrupted for most; maintenance-based replay became the primary mechanism for preserving information, even for high performers.
Retro-Cue Validation: Confirmed that the internal structure is stable and accessible, as post-cue saccades followed the same $1 \to 2 \to 3$ trajectory, validating the existence of an internally generated mental path.

4. Key Results

Pattern Similarity: Gaze trajectories were significantly more similar within the same feature configuration than between different configurations during both encoding and maintenance ( $p < 0.001$ ). Stronger pattern similarity during encoding correlated with higher memory accuracy ( $r = 0.38$ ).
Sequential Structure (3-Item Task):
- Encoding: High performers showed a significant $1 \to 2 \to 3$ sequence strength ( $p < 0.05$ ); low performers did not.
- Maintenance: High performers showed no sequential structure (efficient encoding reduced the need for replay). Conversely, low performers exhibited significant sequential re-visitation ( $1 \to 2 \to 3$ ) during maintenance, suggesting a compensatory mechanism.
Sequential Structure (4-Item Task):
- Increasing load to 4 items disrupted encoding-phase organization for the group.
- During maintenance, high performers (who previously relied on encoding) now showed significant sequential replay, mirroring the strategy of low performers in the 3-item task.
- This indicates that under high cognitive load, the brain shifts the organizational burden from the encoding phase to the maintenance phase.
Retro-Cue: Second saccades after a cue followed the predicted ordinal path ( $1 \to 2$ or $2 \to 3$ ) significantly more often than alternative paths ( $p = 0.003$ ), confirming the stability of the internal sequence.
Control Analyses: Effects were not driven by spatial biases (spatial-only sequence analysis was non-significant) or low-level visual salience (as sequences persisted in the maintenance phase with no visual input).

5. Significance

Bridging Behavior and Neural Mechanisms: The study establishes eye movements as a high-resolution behavioral marker for neural replay. It suggests that just as the hippocampus replays sequences to consolidate weakly encoded memories, the oculomotor system replays sampling sequences to stabilize working memory traces.
Efficient Coding Principle: The findings support the theory that perception and memory share efficient coding principles. The brain actively seeks to impose order (ordinal structure) on chaotic inputs to reduce memory load.
Cognitive Load Adaptation: The research highlights the flexibility of cognitive control. When resources are sufficient, organization happens "online" (encoding); when resources are strained, the system shifts to "offline" reorganization (maintenance replay).
Clinical and Applied Implications: Eye-tracking could serve as a non-invasive biomarker for cognitive load and memory organization deficits. It offers a window into how individuals adaptively restructure information processing strategies under pressure.

In conclusion, this paper redefines eye movements from passive observers of memory to active agents of memory optimization, revealing a flexible, adaptive mechanism where sequential sampling and replay dynamically shift to meet task demands.