SalFormer360: a transformer-based saliency estimation model for 360-degree videos

Technical Summary: SalFormer360

Problem Statement
Saliency estimation is critical for optimizing the transmission and viewing experience of 360-degree (omnidirectional) videos, particularly for tasks like viewport prediction and content optimization. However, existing methods face significant challenges due to the spherical geometry of 360-degree content, which introduces geometric distortions in equirectangular projections. Furthermore, many state-of-the-art approaches rely on computationally expensive techniques, such as spherical convolutions, cubemap projections, or the processing of long temporal sequences (often exceeding 10 frames) and optical flow. These methods often incur high computational overhead and memory requirements, making them less suitable for real-time deployment on resource-constrained devices like Head-Mounted Displays (HMDs). Additionally, accurately modeling human viewing behavior, specifically the tendency to focus on the center of the frame initially (Viewing Center Bias), remains a complex task that many models do not fully address.

Methodology
The authors propose SalFormer360, a lightweight, transformer-based saliency estimation model designed specifically for 360-degree videos. The architecture is built upon the following components:

1. Encoder (SegFormer Backbone): The model utilizes the encoder from SegFormer, a transformer-based architecture originally developed for 2D semantic segmentation. The authors fine-tuned this encoder to handle 360-degree equirectangular frames. To capture temporal dynamics, the model accepts two input frames: the current frame ($t$) and a previous frame ($t-5$). These are concatenated along the channel dimension to form a 6-channel input, with the initial convolutional layer weights adapted to accommodate this.
2. Custom Decoder: A tailored decoder processes the feature maps from the encoder. It consists of a sequence of convolutional layers, batch normalization, ReLU activations, and bilinear upsampling to reconstruct the spatial resolution. A final convolutional layer with a sigmoid activation generates an initial saliency map ($S_{init}$). Dropout (30%) is applied during training to prevent overfitting.
3. Viewing Center Bias (CB) Integration: To account for the human tendency to look at the center of the 360-degree frame upon starting playback, the model incorporates a dynamic Viewing Center Bias. This is achieved by fusing the initial prediction ($S_{init}$) with a pre-computed average saliency map of the first frame (the CB). The fusion is governed by a time-dependent weighting factor ($w_t$) defined as:
   $$S = w_t \cdot CB + (1 - w_t) \cdot S_{init}$$
   The weight $w_t$ combines a static parameter ($\beta_i$) and a decaying dynamic function ($\delta(t) = e^{-\alpha_i(t/C)^2}$). This allows the model to adaptively balance between short-term bias (strong at the start of the video) and long-term bias, with parameters $\alpha_i$ and $\beta_i$ learned during training for each specific dataset.
4. Loss Function: The model is trained using a composite loss function combining four terms: Pearson Linear Correlation Coefficient ($L_{CC}$), Kullback-Leibler divergence ($L_{KL}$), Spherical Mean Squared Error ($L_{SMSE}$), and Binary Cross Entropy ($L_{BCE}$). The $L_{SMSE}$ specifically applies spherical weights to penalize errors near the equator more heavily than those near the poles, addressing projection distortions.

Key Contributions

• Novel Architecture: Introduction of SalFormer360, a transformer-based model that leverages the SegFormer backbone for 360-degree saliency estimation, demonstrating that segmentation architectures can be effectively repurposed for saliency tasks.
• Viewing Bias Modeling: The integration of a learnable Viewing Center Bias mechanism that dynamically adjusts based on frame index and dataset characteristics, improving prediction accuracy by reflecting human exploration behavior.
• Dataset Extension: The authors extended the PVS-HM and VR-EyeTracking datasets by generating ground-truth saliency maps from head-orientation data and extracting RGB frames, making them more accessible for future research.
• Efficiency: The model is designed to be lightweight, utilizing only two frames for temporal context and avoiding complex spherical convolutions or optical flow calculations.

Experimental Results
Extensive experiments were conducted on three large-scale benchmark datasets: Sport360, PVS-HM, and VR-EyeTracking. The model was compared against a wide range of state-of-the-art methods, including 2D video models, 360-degree image models, and 360-degree video models.

• Performance: SalFormer360 outperformed existing methods across all three datasets. In terms of Pearson Correlation Coefficient (CC), it achieved improvements of 8.4% on Sport360, 2.5% on PVS-HM, and 18.6% on VR-EyeTracking compared to the best previously reported results.
• Efficiency: The model is highly efficient, containing only 3.7 million parameters and requiring 14.13 MB of storage. It achieves a computational cost of 1.21 GFLOPs.
• Inference Speed: On an NVIDIA L40s GPU, the model achieves an average inference time of 5.1 ms per frame (approx. 196 fps), confirming its suitability for real-time applications.
• Ablation Studies: Experiments confirmed that both the dynamic ($\delta$) and static ($\beta$) components of the Viewing Center Bias contribute to performance, with their relative importance varying by dataset. The combination of all four loss functions yielded the best overall performance.

Significance and Claims
The paper claims that SalFormer360 demonstrates the representational power of transformer architectures for 360-degree video saliency estimation while maintaining low computational complexity. By successfully adapting a 2D segmentation encoder and integrating a data-driven viewing bias, the model achieves state-of-the-art accuracy without the heavy computational burden of spherical convolutions or long-sequence processing. The authors emphasize that the model's lightweight nature makes it particularly well-suited for edge-side or client-side deployment, such as directly on VR headsets, enabling real-time saliency estimation for viewport prediction and immersive content optimization. The work suggests that accurate, efficient saliency maps can significantly enhance the streaming efficiency of 360-degree videos by allowing high-quality transmission of salient regions while downgrading peripheral areas.