🤖 machine learning

Consensus Clustering of Free-Viewing Gaze Data: New Insights into Human-Information Interaction

This paper introduces EnsembleGaze, a novel unsupervised ensemble learning system that applies consensus clustering to free-viewing gaze data, revealing robust distinctions in ambient versus focal viewing modes for stimuli while demonstrating that user behavior patterns are context-dependent and best captured through biclustering strategies.

Original authors: Beryl Gnanaraj, Jaya Sreevalsan-Nair, Saqib Alam Ansari, Maanasa Rajaraman

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Beryl Gnanaraj, Jaya Sreevalsan-Nair, Saqib Alam Ansari, Maanasa Rajaraman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a group of people look at a series of pictures on a wall. You can't hear what they are thinking, but you have a special camera that tracks exactly where their eyes land, how long they stare, and how their eyes jump from one spot to another. This is gaze data.

For a long time, scientists have tried to make sense of this data by looking at individual "fixations" (where the eye stops) or "saccades" (the jumps between stops). But the authors of this paper felt this was like trying to understand a whole orchestra by only listening to one instrument at a time. They wanted to hear the whole symphony: how the people interact with the pictures together.

Here is a simple breakdown of their new system, EnsembleGaze, and what they discovered.

The Problem: Too Many Ways to Slice the Pie

When you try to group people based on how they look at pictures, there are many ways to do it.

You could group them by how fast their eyes move.
You could group them by how long they stare.
You could group them by the colors in the pictures.

The problem is that if you pick just one way to group them, you might get a biased result. It's like asking a group of friends to sort a deck of cards. If you ask them to sort by color, you get one pile. If you ask them to sort by number, you get a different pile. Neither is "wrong," but neither tells the whole story.

The Solution: The "Super-Group" (Consensus Clustering)

To solve this, the authors built a system called EnsembleGaze. Think of this system as a super-judge panel.

Instead of relying on one method to sort the data, they asked several different "sorting algorithms" (mathematical methods) to do the job independently.

The Panel: They used three different "judges" (algorithms) to look at the data.
The Vote: Each judge creates their own groups. Then, the system looks at all the votes. If Judge A, Judge B, and Judge C all agree that Person X and Person Y belong in the same group, the system is very confident that they should be together.
The Final Decision: The system creates a "Consensus" (a final agreement) based on these votes. This reduces the chance of error and gives a much more reliable picture of who belongs with whom.

The Twist: Looking at Two Things at Once (High-Dimensional Clustering)

Most studies look at either the People OR the Pictures.

Study A: "Which people look at pictures the same way?"
Study B: "Which pictures attract the same kind of attention?"

The authors wanted to do both at the same time. They treated the combination of (Person + Picture) as a single unit. Imagine a dance floor where you are trying to find pairs that dance well together. You aren't just grouping the dancers, and you aren't just grouping the songs; you are grouping the specific dance moves that happen when a specific person hears a specific song.

They used two advanced techniques for this:

Subspace Clustering: This is like sorting the dancers first by their style, and then, within each style group, sorting them by the song they are dancing to. It's a step-by-step approach.
Biclustering: This is like sorting the dancers and the songs simultaneously. It finds a specific group of people who all love a specific set of songs, while ignoring the rest.

What They Found (The Results)

They tested this system on two famous sets of eye-tracking data: one with natural scenes (like landscapes) and one with emotional images.

1. The "Ambient" vs. "Focal" Split
They found that pictures naturally fell into two main groups, regardless of the method used:

The "Ambient" Group: These are pictures where people's eyes wander around broadly, taking in the whole scene (like looking at a landscape).
The "Focal" Group: These are pictures where people lock onto specific details (like a face or a car).
The Takeaway: The pictures themselves have a "personality" that dictates how we look at them.

2. People are Context-Dependent
The groups of people were not fixed. A person might look like a "Focal" watcher when looking at a landscape, but an "Ambient" watcher when looking at an emotional image.

The Takeaway: You can't just label a person as "a fast looker" or "a slow looker." Their behavior changes depending on what they are looking at. Only the advanced "two-at-once" methods (Biclustering) could find this complex pattern.

3. Colors Don't Matter as Much as You Think
The authors checked if the color of the picture (Red vs. Blue) or how bright it was (Light vs. Dark) dictated how people looked at it.

The Takeaway: Surprisingly, no. The colors and brightness of the images did not strongly predict how people grouped together.

4. Objects Matter More
However, what was in the picture mattered a lot.

If a picture had a lot of people or wildlife, it created distinct viewing patterns.
If a picture had objects (like cars or furniture), it created different patterns.
The Takeaway: The content (the objects) drives the behavior, not just the colors.

The Bottom Line

The authors built a tool (EnsembleGaze) that acts like a smart, multi-voting system to understand how humans interact with images. They discovered that:

Pictures naturally split into "broad view" and "close-up view" categories.
People don't have one fixed way of looking; they change their style based on the picture.
What is in the picture (objects) matters more than the colors.

This system provides a reliable, repeatable way to study human attention without needing to guess the "right" answer beforehand. It's a new way to listen to the "symphony" of human attention rather than just listening to one instrument.

Technical Summary: Consensus Clustering of Free-Viewing Gaze Data

Problem Statement
Free-viewing gaze data offers a rich, task-free window into human visual attention. While conventional exploratory data analysis (EDA) typically extracts user attention patterns through fixations and Areas of Interest (AOIs), the deeper Human-Information Interaction (HII) patterns within this data remain understudied. Traditional clustering of gaze data often relies on specific criteria (e.g., fixation duration or saccade patterns) without aggregating user behavior and image stimulus characteristics simultaneously. Furthermore, existing clustering approaches suffer from modeling bias and uncertainty in validation due to the lack of ground truth. There is a specific gap in integrating user interaction with contextual visual features to understand how image characteristics influence user behavior in vision tasks.

Methodology: EnsembleGaze
To address these gaps, the authors propose EnsembleGaze, a novel end-to-end unsupervised ensemble learning system designed for the consensus clustering of free-viewing gaze datasets. The system operates through the following pipeline:

Feature Engineering:
- Fixation-based Features: The system extracts statistical descriptors (total, mean, median, standard deviation, variance) from distributions of total fixation counts (FC), time-to-first-fixation (TTF), and fixation durations (FD). These are aggregated into feature vectors for users, stimuli, or (user, stimulus) tuples.
- Scanpath-Distance Features: For user clustering, geometric distances between scanpaths are computed using metrics such as Fréchet, Hausdorff, Euclidean, Dynamic Time Warping (DTW), and String-based Time Delay Embedding (SBTDED).
- Image Property Categorization: Stimuli are categorized based on color (hue and intensity via histograms) and object properties (detected via YOLO, categorized by type, scale, and color).
Consensus Clustering (Baseline):
- The system employs an unsupervised ensemble learning paradigm where multiple base clustering algorithms (k-means++, hierarchical agglomerative, and spectral clustering) are applied to the feature vectors.
- Co-association Matrix: A binary co-association matrix is generated for each base clustering, indicating pairwise membership.
- Consensus Voting: These matrices are averaged to create a probabilistic co-association matrix.
- Meta-Learning: Hierarchical Agglomerative Clustering (HAC) is applied to the consensus matrix (via a dissimilarity matrix) to determine the final optimal partitioning and the number of clusters ( $k$ ).
High-Dimensional Clustering:
To analyze the joint influence of users and stimuli, the system implements two strategies:
- Consensus Subspace Clustering (CSC): A sequential, top-down approach. It first clusters one dimension (e.g., users) and then clusters the dual dimension (e.g., stimuli) within those primary clusters, using consensus voting at each step.
- Spectral Biclustering: A simultaneous clustering method that groups (user, stimulus) tuples by identifying submatrices in the feature tensor. This involves matrix unfolding (mode-1 for user-guided, mode-2 for stimulus-guided) followed by spectral decomposition and k-means on the eigenvectors.
Validation:
Since ground truth is absent, the system evaluates performance using standard metrics (Silhouette index, Davies-Bouldin, Calinski-Harabasz) and comparative metrics (Cohen's $\kappa$ , Accuracy, MCC, F1-score, ARI, NMI) against baseline clusterings.

Key Contributions

EnsembleGaze System: An end-to-end framework for unsupervised ensemble learning of free-viewing gaze data, integrating feature engineering, consensus clustering, and high-dimensional clustering.
Consensus Clustering Strategy: The application of consensus voting via co-association matrices to reduce modeling bias and improve the robustness of clustering outcomes in the absence of ground truth.
High-Dimensional Strategies: The adaptation of Consensus Subspace Clustering and Spectral Biclustering to gaze data to capture the joint (user, stimulus) dynamics, moving beyond one-way clustering.
Feature Engineering: A systematic approach using statistical descriptors of fixation distributions and image/object properties to characterize user behavior and stimulus types.

Results
The system was evaluated on two public datasets: MIT1003 (natural scenes) and EMOd (emotion-eliciting images).

Clustering Performance:
- MIT1003: Image stimuli groupings were highly consistent across methods, reflecting a robust distinction between "ambient" (broad) and "focal" (specific) viewing modes. User groupings were found to be image-context-dependent. The Hausdorff distance metric yielded the best results for user clustering based on scanpath distances.
- EMOd: Dataset-specific patterns emerged. The EMOd dataset showed fixation patterns that were more linearly separable (aligning well with KMeans++) compared to the non-linear separability of MIT1003 (aligning with Spectral clustering).
High-Dimensional Insights:
- Stimulus-Guided Clustering: In MIT1003, stimulus-guided CSC and biclustering showed high alignment, identifying a large partition driven by high fixation counts (ambient attention) and a smaller partition driven by fixation duration variance (focal attention). In EMOd, stimulus-guided CSC revealed partitions characterized by erratic fixation durations, potentially linked to emotional distress or cognitive overload, a pattern less captured by biclustering.
- User-Guided Clustering: User groupings were less separable in EMOd, with some configurations failing to produce distinct clusters, suggesting that fixation features alone may not be sufficient to separate users in all contexts.
Feature Influence:
- Image-level color hue and intensity showed weak correlation with user fixation patterns.
- Object-property-based features (e.g., object scale, type) yielded more robust clustering partitions, evidenced by distinct block-diagonal patterns in co-association matrices.

Significance and Claims
The paper claims that EnsembleGaze provides a replicable, dataset-agnostic methodology for the unsupervised analysis of fixation behavior in scene perception research. The authors assert that their results demonstrate:

Robustness: Image stimuli groupings are consistent across methods, validating the existence of distinct viewing modes (ambient vs. focal).
Context-Dependency: User groupings are not static but depend on the image context; only biclustering and two-step conditional approaches (CSC) are architecturally capable of recovering this structure.
Methodological Advancement: The study establishes that unsupervised ensemble learning and high-dimensional clustering (specifically biclustering and subspace clustering) are necessary to uncover latent HII patterns that single-metric or one-way clustering misses.
Feature Relevance: The findings suggest that object-level properties are more predictive of clustering structure than simple color or intensity metrics.

The authors maintain a modest tone regarding the limitations, noting that the study is confined to fixation-based features and that the non-separability of certain user clusters in EMOd may indicate a need for additional features (e.g., saccades) or data-driven feature learning in future work.

The Problem: Too Many Ways to Slice the Pie

The Solution: The "Super-Group" (Consensus Clustering)

The Twist: Looking at Two Things at Once (High-Dimensional Clustering)

What They Found (The Results)

The Bottom Line

More like this