Can Individual Internal Models Predict Idiosyncratic Scene Exploration?

This study demonstrates that individual differences in internal models of typical scenes predict unique gaze exploration patterns, but only when strategic, expectation-guided viewing is encouraged rather than during free viewing.

The Gist

The Big Question: Why Do We Look at Things Differently?

Imagine two people walking into the same messy kitchen.

• Person A immediately looks at the sink to see if the dishes are clean.
• Person B immediately looks at the fridge to see if there's food.
• Person C looks at the floor to see if it's wet.

Even though they are looking at the exact same room, their eyes go to different places. Scientists call this idiosyncratic gaze behavior. We know it happens, but why? Is it just random? Is it because Person A is hungry and Person B is thirsty?

This paper asks: Is it because our brains have different "mental blueprints" of what a kitchen is supposed to look like?

The Hypothesis: The "Mental Blueprint" Theory

The researchers believe that everyone carries an Internal Model (or a mental blueprint) in their head. This is your brain's expectation of how the world usually looks.

• If your mental blueprint of a kitchen says, "The sink is usually on the left," your eyes will likely go there first.
• If your blueprint says, "The fridge is the most important thing," you'll look there first.

The researchers wanted to know: If two people have very similar mental blueprints, will they look at the world in the same way?

The Experiment: Drawing vs. Watching

To test this, the researchers did a two-part experiment with two groups of people.

Part 1: The "Draw Your Dream" Test

First, they asked participants to draw a typical bathroom or kitchen. They didn't ask for art; they just wanted to see what the participants' brains thought was "normal."

• The Analogy: Imagine asking everyone to draw a map of their hometown from memory. If two people draw a map where the bakery is always on the corner, they probably share a similar mental map of that town.

They used a super-smart computer (AI) to compare these drawings. It measured how similar Person A's drawing was to Person B's drawing. This created a "Similarity Score" for their Internal Models.

Part 2: The Eye-Tracking Test

Next, they showed these same people photos of real kitchens and bathrooms and tracked their eye movements. They measured:

1. Where they looked.
2. How many times they stopped to look (fixations).
3. The order in which they looked at different things (e.g., sink first, then stove).

The Twist: Two Different Worlds

The researchers ran this test in two different ways to see if the "mental blueprint" mattered more in some situations than others.

Experiment 1: The "Free Roam" (Easy Mode)

• The Setup: People looked at clear, high-quality photos of kitchens for 3 seconds. They could see everything at once.
• The Result: No connection found. Even though everyone had unique eye movements, those movements didn't match their drawings.
• Why? When everything is clear and easy to see, our eyes are lazy. We just look at what is bright, colorful, or obvious (like a shiny toaster). We don't need to use our "mental blueprints" because the answer is right in front of us.

Experiment 2: The "Foggy Window" (Hard Mode)

• The Setup: This time, the researchers made it hard. They put a "foggy window" over the screen. You could only see a tiny, clear circle around where your eye was looking. The rest of the image was blurry and gray.
• The Task: People had to memorize the scene for a memory test.
• The Result: A strong connection found! Now, the people who had similar drawings (similar mental blueprints) also looked at the scene in very similar ways.
• Why? Because the image was blurry, they couldn't just "see" where to look. They had to guess. They had to use their internal blueprint to say, "I bet the sink is over there," and move their eyes to check. When the world is unclear, our expectations take the wheel.

The Takeaway: When Do Our Expectations Rule?

Think of your eyes like a car and your brain like the driver.

• In Experiment 1 (Clear View): The road is sunny and clear. The car drives itself (bottom-up processing). The driver (your expectations) doesn't need to do much. Everyone drives a bit differently, but it's not because of their map; it's just because they are different drivers.
• In Experiment 2 (Foggy View): The road is covered in thick fog. The car's sensors can't see the road. Now, the driver must rely on their mental map to navigate. If two drivers have the same mental map of the city, they will take the exact same turns through the fog.

Conclusion

The study shows that our personal expectations (internal models) only strongly guide our eyes when we are unsure of what we are seeing.

When the world is clear, we just look at what's shiny. But when the world is confusing, blurry, or we are trying to solve a puzzle, we rely on our unique "mental blueprints" to decide where to look next. This explains why, in difficult situations, we all explore the world in our own unique ways based on what we expect to find.

Technical Summary

1. Problem Statement

While it is well-established that individuals exhibit systematic differences in how they explore natural scenes (e.g., gaze dynamics, spatial biases, and semantic category salience), the underlying causes of these idiosyncrasies remain poorly understood. Previous research suggests gaze behavior is shaped by accumulated visual experience ("visual diet"), which theoretically gives rise to internal models—expectations about how specific environments typically look.

The central hypothesis of this study is that inter-subject variations in internal models predict inter-subject variations in gaze behavior. Specifically, individuals with more similar internal representations of a scene category (e.g., a kitchen) should exhibit more similar eye movement patterns when viewing real-world images of that category. The study aims to test this hypothesis and determine under what conditions (e.g., free viewing vs. strategic exploration) these internal models influence gaze.

2. Methodology

The researchers employed an Inter-Subject Representational Similarity Analysis (IS-RSA) framework across two experiments involving independent groups of participants ($N=34$ per experiment).

A. Quantifying Internal Models (The Predictor)

• Task: Participants drew their "most typical" examples of two scene categories: bathrooms and kitchens.
• Control: To account for individual drawing styles or motor abilities, participants also copied identical photographs of scenes.
• Feature Extraction:
  • Drawings were converted into photorealistic images using commercial Draw3D software.
  • These images were processed by a deep neural network (VGG-16, pre-trained on ImageNet).
  • Feature activations from the late layer (fc7) were extracted.
  • IS-RDM Construction: Pairwise correlations of these feature vectors between participants created an Inter-Subject Representational Dissimilarity Matrix (IS-RDM) representing the similarity of their internal models. The control condition IS-RDM was partialled out in subsequent analyses to isolate semantic expectations from stylistic variance.

B. Quantifying Gaze Behavior (The Outcome)

Participants viewed 300 real-world scene photographs (150 kitchens, 150 bathrooms) for 3 seconds each. Gaze was recorded at 120 Hz. Four distinct gaze-based IS-RDMs were constructed to capture different aspects of exploration:

1. Gaze Distance: Median Euclidean distance between the gaze positions of participant pairs.
2. Fixation Count: Pairwise correlations of the number of fixations per image.
3. Object Dwells: Pairwise correlations of proportional dwell times on specific object categories (derived from LabelMe annotations).
4. Fixation Order: Pairwise correlations of "priority scores" (rank-ordered time of first fixation) for object categories.

C. Experimental Conditions

• Experiment 1 (Free Viewing): Participants viewed full-resolution images without specific task constraints.
• Experiment 2 (Gaze-Contingent Viewing):
  • Sensory Uncertainty: Only a small circular window ($2.5^\circ$ radius) around the current gaze was shown in full resolution; the rest of the image was heavily degraded (Gaussian smoothed and desaturated).
  • Strategic Demand: Participants performed a subsequent memory task, encouraging them to use prior knowledge to guide sampling.

D. Statistical Analysis

• IS-RSA: Partial correlations were computed between the Internal Model IS-RDM and each of the four Gaze IS-RDMs.
• Significance: Assessed via non-parametric subject-wise permutation tests (10,000 iterations) with False Discovery Rate (FDR) correction.
• Reliability: Split-half reliability analyses were conducted to ensure individual differences in gaze were stable across stimulus subsets.

3. Key Results

Reliability

Both experiments demonstrated reliable individual differences in all four gaze measures (split-half correlations were significant, $p < 0.001$), confirming that the metrics successfully captured idiosyncratic viewing patterns.

Experiment 1: Free Viewing

• Result: No significant correlation was found between inter-subject similarities in internal models and any of the gaze measures (all $p > 0.65$).
• Implication: Under unconstrained viewing with full sensory information, individual expectations do not significantly predict how people explore a scene.

Experiment 2: Gaze-Contingent Viewing

• Result: Significant correlations emerged between internal model similarity and gaze behavior:
  • Fixation Count: $r = 0.21, p = 0.025$.
  • Fixation Order: $r = 0.15, p = 0.025$.
• Null Results: Gaze distance and object dwell times remained uncorrelated with internal models.
• Implication: When sensory input is restricted and a strategic task (memory) is imposed, individuals rely more heavily on their internal models to guide where and in what order they sample information.

Drawing Order vs. Fixation Order

• The order in which participants drew objects did not significantly predict their own fixation order compared to others ($\Delta r = 0.03, p = 0.081$), suggesting that drawing strategies may not perfectly map to perceptual priorities in this context.

4. Key Contributions

1. Conditional Influence of Priors: The study provides empirical evidence that internal models drive gaze behavior primarily under conditions of high sensory uncertainty and strategic demand. In rich, unconstrained environments, bottom-up saliency or other subject-specific traits (e.g., aesthetic preference) may dominate.
2. Methodological Innovation: The paper successfully integrates deep learning pipelines (VGG-16 on AI-generated photorealistic drawings) with IS-RSA to quantify abstract internal models objectively, moving beyond self-report or simple behavioral proxies.
3. Granular Gaze Metrics: The study demonstrates that different gaze metrics capture different cognitive processes. Internal models predicted the strategy of exploration (fixation count and order) but not the spatial trajectory (gaze distance) or duration (dwell time).

5. Significance and Conclusion

This research advances the predictive processing framework of vision by demonstrating that eye movements are not merely stimulus-driven but reflect an active sampling process guided by individual expectations.

• Theoretical Impact: It resolves the debate on whether gaze is driven by bottom-up saliency or top-down expectations by showing that top-down internal models become the dominant driver when bottom-up information is insufficient to resolve uncertainty.
• Practical Application: The findings suggest that computational models of attention and eye movement must incorporate idiosyncratic internal models to accurately predict human behavior in complex, real-world scenarios, particularly when visual information is degraded or task demands are high.
• Future Directions: The authors suggest that future work should refine how internal models are captured (e.g., constraining drawing time) to better align with perceptual priorities and explore how these models function in other domains of visual cognition.