Judging the reasons for fixations: A direct experimental method to assess the contribution of saliency and semantic factors to gaze control

Technical Summary: Judging the Reasons for Fixations

Problem Statement
Current debates concerning the relative contributions of low-level visual saliency versus high-level semantics in gaze control rely heavily on indirect, global approaches. These methods typically compare the predictive power of saliency maps against meaning maps across entire scenes. The authors argue that such global comparisons are fundamentally limited because they conflate heterogeneous, local causes of fixations. By aggregating data over whole scenes, these approaches fail to distinguish the specific, immediate drivers behind individual fixation decisions.

Methodology
To address the limitations of indirect comparison, the study employs a direct experimental method. Participants were presented with visual stimuli and asked to explicitly identify the reasons for fixations at specific clusters of high fixation density. Rather than inferring causes from statistical correlations, participants directly distinguished between:

• Low-level saliency.
• Various semantic categories.
• The most important factor driving the fixation.

This approach allows for a granular analysis of the specific motivations behind gaze behavior at localized points of interest.

Key Results
The judgments obtained from participants revealed that gaze control is driven by multiple factors acting simultaneously. Key findings include:

• Dominance of Semantics: While the influence of factors varied across different fixation clusters, semantic factors generally dominated low-level saliency.
• Role of Abstract Semantics: Abstract semantic categories, particularly those labeled "unknown/unusual," were found to be highly important. This highlights that attention is guided not only by personal relevance but also by prior knowledge and novelty.
• Model Performance Discrepancy: The study compared the predictive performance of classic saliency and meaning maps against deep learning-based models (specifically DeepGaze IIE). It was observed that the predictive performance of DeepGaze IIE varied less significantly depending on the specific reasons for fixation compared to the classic map-based approaches.

Key Contributions and Proposed Framework
The paper proposes a conceptual framework to interpret these findings within the context of existing gaze control models. This framework distinguishes between two distinct processes:

1. Highlighting: Processes that identify interesting locations within an image.
2. Sampling Strategy: The mechanism that translates these highlighted locations into actual scanpaths.

Within this framework:

• Classic saliency and meaning maps are viewed as restricted inputs that primarily feed into the "highlighting" process.
• Deep learning-based models (e.g., DeepGaze IIE) are characterized as more general; they may implicitly encode aspects of the "sampling strategy" itself, in addition to processing image features. This structural difference explains why deep learning models demonstrate more robust predictive performance across varying fixation reasons compared to traditional map-based approaches.

Significance
The significance of this work lies in its methodological shift from indirect, global correlation to direct, local attribution of fixation causes. By demonstrating that semantics generally outweigh saliency and that abstract novelty plays a critical role, the study challenges the sufficiency of current global comparison methods. Furthermore, by proposing a framework that separates image highlighting from scanpath strategy, the paper offers a new lens for understanding why deep learning models outperform classic approaches, suggesting they capture a more comprehensive representation of the gaze control process.