💻 computer science

MASS-ScanDMM: A Panoramic Image Scanpath Prediction Method Integrating Multiple Attention Mechanisms and Spherical Semantic Enhancement

This paper proposes MASS-ScanDMM, a panoramic image scanpath prediction method that integrates Multiple Attention Scene Perception (MASP) and Spherical Information Enhancement Perception (SIEP) mechanisms to achieve superior accuracy and generalization across multiple datasets.

Original authors: Jianwei Li, Yuxin Chen, Hongjue Chen, Sixi Chen

Published 2026-09-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jianwei Li, Yuxin Chen, Hongjue Chen, Sixi Chen

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital age, our screens are no longer flat rectangles but immersive windows into 360-degree worlds. Virtual reality headsets allow us to look in any direction, turning a simple image into a vast, spherical environment where we can turn our heads to explore a museum, a park, or a living room. However, rendering these entire worlds in high definition is incredibly demanding on computer power and internet bandwidth. To solve this, researchers look to how human eyes actually work. Our vision is not uniform; we see fine detail only in a tiny central spot, while the rest of our view is blurry. By predicting where a person will look next, computers can render only that small, sharp area in high definition and keep the rest low-resolution, saving massive amounts of resources. This prediction of eye movement, known as a scanpath, is the key to making virtual reality efficient and responsive.

For decades, scientists have studied how people scan flat, two-dimensional pictures, but the rules change completely when the image wraps around a sphere. In a flat photo, the eye naturally gravitates toward the center, but in a 360-degree panorama, the viewer must actively turn their head to find interesting details. Existing computer models often struggle with this, producing gaze patterns that are either too random or fail to account for the unique geometry of a spherical world. They might miss important objects or suggest eye movements that feel unnatural to a human observer. The challenge lies in teaching a computer to understand not just the objects in a scene, but how those objects exist within a curved, all-encompassing space.

A team of researchers from Fuzhou University has addressed this challenge with a new method called MASS-ScanDMM. Their approach is designed to mimic the complex way human brains process visual information in these immersive environments. Instead of relying on a single way of looking at an image, the new system combines several different strategies to decide where a viewer's eyes will go. It treats the panoramic image as a dynamic scene where attention shifts over time, much like a person exploring a room. The system is built on a foundation that tracks visual states, essentially simulating a form of working memory that holds onto what has just been seen to predict what will be seen next.

The core of this new method involves two specialized components that work together to improve accuracy. The first component acts as a sophisticated filter, helping the computer decide which parts of the image are most important. It does this by weighing different aspects of the visual data simultaneously, such as the brightness of specific areas, the texture of objects, and the relationships between different parts of the scene. By integrating these multiple layers of attention, the system becomes much better at identifying the specific spots that will catch a human eye, rather than getting lost in the vastness of the 360-degree view.

The second component focuses specifically on the unique shape of the panoramic image. Because a 360-degree photo is often stretched out on a flat screen for computers to process, important spatial relationships can get distorted. This new system corrects for that distortion by grouping information in a way that respects the spherical nature of the world. It analyzes the image by looking at horizontal and vertical directions separately, ensuring that the computer understands how the top of a building connects to the bottom, even when the image is wrapped around. This allows the model to maintain a clear sense of the scene's structure, preventing the confusion that often plagues other methods when dealing with the edges of a panoramic view.

When the researchers tested their system against existing models using three different large collections of panoramic images, the results were clear. They measured how closely the computer's predicted eye movements matched the actual paths taken by real human viewers. The new method consistently outperformed previous approaches, generating gaze patterns that were smoother and more accurate. In one set of tests involving a museum scene, the system successfully predicted that viewers would look at specific exhibits, whereas older models often scattered their predictions across empty walls. The data showed that the new system reduced the error in prediction significantly, bringing the computer's behavior much closer to the natural way humans explore these environments.

The researchers also found that the system could be used for a related task: creating heat maps that show which parts of a scene are most likely to attract attention. By aggregating the predicted eye movements, they could generate a visual guide that highlighted the most interesting areas of a panoramic image. This application proved that the system not only predicts movement but also understands the content of the scene well enough to identify what makes a place compelling. The study confirms that by combining multiple ways of paying attention with a deep understanding of spherical geometry, computers can finally learn to see the world the way we do, opening the door to more efficient and realistic virtual reality experiences.

Technical Summary: MASS-ScanDMM

Problem Statement

The paper addresses the challenge of predicting human scanpaths (gaze trajectories) in 360° panoramic images, a critical task for optimizing Virtual Reality (VR) rendering through foveated rendering and enhancing human-computer interaction. While scanpath prediction is well-studied for 2D images, panoramic images present unique difficulties due to their 360°×180° field of view, complex spatial dependencies, and the variability of user behavior when exploring content via Head-Mounted Displays (HMDs).

Existing methods for 360° scanpath prediction suffer from specific limitations:

Saliency-map-based approaches rely heavily on low-level visual features and struggle to model the randomness of human gaze.
Generative models often produce unstable paths due to reliance on data fitting.
Probabilistic models, such as the Deep Markov Model (ScanDMM), improve temporal dependency modeling but exhibit insufficient capability in extracting spherical scene semantics, leading to deviations between predicted and actual gaze trajectories.

Methodology

The authors propose MASS-ScanDMM (Multi-Attention Spherical Semantic Deep Markov Model), a unified framework designed to enhance both spatial feature extraction and temporal attention modeling in spherical scenes. The core of the method is the MAS-SSEPNet (Multi-Attention Spherical Scene Semantic Enhancement Perception Network), which feeds semantic features into a Deep Markov Model to generate realistic scanpath sequences.

The network integrates two novel mechanisms:

1. Multiple Attention Scene Perception (MASP) Mechanism

Designed to overcome the limitations of conventional attention mechanisms in complex scenes, MASP dynamically adjusts feature map response intensity by integrating five complementary attention types:

Channel Attention: Reweights input feature maps to emphasize informative channels.
Kernel Attention: Selects or combines multiple convolution kernels to adapt to diverse features.
Kernel Space Attention: Adjusts spatial sensitivity within convolution kernels, amplifying critical positions.
Filter Attention: Controls the importance of output filters to suppress redundant responses.
Neuron Attention: Evaluates individual neuron deviations from the global mean to generate pixel-level attention weights, enhancing local saliency.

These weights are fused to create a multi-dimensional perception network that refines feature representations and directs focus toward key visual regions.

2. Spherical Information Enhancement Perception (SIEP) Mechanism

To address the geometric distortions inherent in spherical images, SIEP employs a grouped channel decomposition strategy combined with directional pooling:

Grouped Processing: Input features are divided into groups to avoid information loss from global compression.
Directional Pooling: Parallel branches process features using 1×1 and 3×3 convolutions. The 1×1 branch performs horizontal and vertical average pooling to extract global context, generating global weights ( $W_h, W_w$ ).
Adaptive Fusion: The mechanism uses Group Normalization and adaptive global pooling to fuse global context with local geometric correlations. It computes pixel-level associations to dynamically emphasize or suppress regions based on learned attention weights, preserving spherical geometry while enhancing semantic representation.

Key Contributions

The paper outlines four primary contributions:

Framework Proposal: Introduction of MASS-ScanDMM, a novel deep probabilistic framework that explicitly models temporal visual states and spherical scene semantics for 360° scanpath prediction.
MASP Module: Design of a module integrating neuron, filter, kernel, spatial, and channel attention to enhance feature responses in complex visual scenes.
SIEP Module: Development of a mechanism using grouped channel decomposition and directional pooling to strengthen spherical semantic representations and improve feature utilization.
Empirical Validation: Extensive experiments on three datasets (Sitzmann, Salient360!, and JUFE) demonstrating the model's superior accuracy and generalization compared to state-of-the-art baselines.

Experimental Results

The authors evaluated MASS-ScanDMM against four baseline methods (DeepGaze III, SaltiNet, ScanGAN, and ScanDMM) using three standard metrics: Levenshtein distance (LEV), Dynamic Time Warping (DTW), and Cross-Recurrence (REC).

Quantitative Performance: On the Sitzmann dataset, MASS-ScanDMM achieved the best scores among all models (excluding human benchmarks) with an LEV of 46.280, DTW of 1937.367, and REC of 3.398. Similar superior performance was observed on the Salient360! and JUFE datasets.
Ablation Studies: Experiments confirmed the individual and combined contributions of MASP and SIEP. Adding MASP reduced LEV by 0.703 and DTW by 15.576 compared to the baseline. Adding SIEP reduced LEV by 1.193 and DTW by 89.188. The full model outperformed the partial configurations, validating the complementary nature of the modules.
Qualitative Analysis: Visualizations showed that MASS-ScanDMM generates smoother scanpaths that align more closely with human fixation patterns, particularly in salient regions, whereas other methods exhibited large displacements or sparse fixation points.

Application to Saliency Detection

The model was also applied to saliency detection tasks on the Salient360! dataset. MASS-ScanDMM achieved the best performance in Area Under the Curve (AUC) and Kullback-Leibler Divergence (KLD), and competitive results in Normalized Scanpath Saliency (NSS) and Correlation Coefficient (CC), outperforming dedicated 360° saliency detection methods like BMS360 and GBVS360.

Significance and Claims

The paper claims that MASS-ScanDMM represents a significant advancement in panoramic image analysis by effectively bridging the gap between temporal visual state modeling and spherical semantic understanding. By integrating multi-dimensional attention mechanisms with geometry-aware representations, the method successfully captures salient regions and potential points of interest more accurately than existing approaches.

The authors conclude that the proposed approach validates the effectiveness of jointly modeling cross-attention and spherical feature extraction. While acknowledging a remaining performance gap between computational models and human upper bounds, the study highlights the method's potential for broader applications in panoramic visual understanding, particularly in optimizing VR rendering and supporting user intent inference.