💻 computer science

NTIRE 2026 Challenge on Video Saliency Prediction: Methods and Results

This paper presents the NTIRE 2026 Challenge on Video Saliency Prediction, detailing the creation of a novel open-license dataset with over 5,000 assessors, the participation of 20 teams, and the evaluation of seven final methods on 800 test videos, with all resources made publicly available.

Original authors: Andrey Moskalenko, Alexey Bryncev, Ivan Kosmynin, Kira Shilovskaya, Mikhail Erofeev, Dmitry Vatolin, Radu Timofte, Kun Wang, Yupeng Hu, Zhiran Li, Hao Liu, Qianlong Xiang, Liqiang Nie, Konstantinos Ch

Published 2026-04-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Andrey Moskalenko, Alexey Bryncev, Ivan Kosmynin, Kira Shilovskaya, Mikhail Erofeev, Dmitry Vatolin, Radu Timofte, Kun Wang, Yupeng Hu, Zhiran Li, Hao Liu, Qianlong Xiang, Liqiang Nie, Konstantinos Chaldaiopoulos, Niki Efthymiou, Athanasia Zlatintsi, Panagiotis Filntisis, Katerina Pastra, Petros Maragos, Li Yang, Gen Zhan, Yiting Liao, Yabin Zhang, Yuxin Liu, Xu Wu, Yunheng Zheng, Linze Li, Kun He, Cong Wu, Xuefeng Zhu, Tianyang Xu, Xiaojun Wu, Wenzhuo Zhao, Keren Fu, Gongyang Li, Shixiang Shi, Jianlin Chen, Haibin Ling, Yaoxin Jiang, Guoyi Xu, Jiajia Liu, Yaokun Shi, Jiachen Tu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are standing in a busy, chaotic square. There are people shouting, cars honking, birds flying, and a street performer juggling. Your brain doesn't try to process everything at once. Instead, it instantly zooms in on the most interesting things: the juggler, a flashing neon sign, or a sudden movement. This natural "zooming in" is called visual saliency.

This paper is about a high-stakes competition (the NTIRE 2026 Challenge) where computer scientists tried to teach computers to do exactly what your brain does: figure out where a human is looking in a video, frame by frame.

Here is the story of how they did it, explained simply.

1. The Problem: Teaching a Robot to "Look"

For a long time, computers have been great at recognizing what is in a picture (e.g., "That's a dog"). But they are terrible at understanding where a human would look at that dog. Is it the dog's tail? Its eyes? The ball it's chasing?

To solve this, the organizers needed a massive library of videos where they knew exactly where thousands of people looked.

The Old Way: Using expensive eye-tracking glasses on volunteers. This is like hiring a personal photographer for every single person watching a movie. It's slow and expensive.
The New Way (Used here): They used a clever trick. They asked over 5,000 regular people to watch videos on their computers and move their mouse cursors as if they were looking at the screen.
The Result: They built a massive dataset of 2,000 videos with over 1 million frames. They treated the mouse movements like a "digital eye," creating a map of where people were most likely to look.

2. The Competition: The "Eye-Tracking Olympics"

Once they had the data, they held a contest. Over 20 teams of researchers and students from around the world entered. They had to build AI models that could look at a video and draw a "heat map" showing where a human would look.

The judges used four different rulers to measure how good the AI was:

Correlation: How well does the AI's guess match the real human map?
Similarity: Do the shapes of the "hot spots" look the same?
Area Under the Curve: Did the AI pick the right spots more often than a random guess?
Scanpath: Did the AI follow the same path as the human eyes?

3. The Winners: How the Top Teams Did It

Seven teams made it to the final round. Here is how the top teams built their "super-eyes," using some creative metaphors:

🥇 1st Place: The "Double-Brain" Team (iLearn)
- The Idea: They didn't trust just one way of thinking. They built an AI with two experts working together.
- Expert 1 is like a detective who knows that humans usually look at the center of the screen first.
- Expert 2 is a wild card that looks for weird, off-center things.
- The Magic: They let both experts make a guess, then averaged their answers. It's like asking two different detectives to solve a crime and taking the consensus. This worked best.
🥈 2nd Place: The "Time-Traveler" Team (CVSP)
- The Idea: This team used a model trained to predict the future.
- The Metaphor: Imagine watching a movie and guessing what happens next. If you guess wrong, your brain pays extra attention to that moment because it was surprising. This team taught their AI to predict what the next frame of a video would look like. When the AI got "surprised" (made a prediction error), it marked that spot as "salient" (important).
🥉 3rd Place: The "Layer Cake" Team (ARK MMLAB)
- The Idea: They built a pyramid of information.
- The Metaphor: They looked at the video in layers. The bottom layer sees tiny details (like fur texture), while the top layer sees the big picture (like "a dog running"). They mixed all these layers together so the AI could see both the forest and the trees at the same time.
Other Cool Approaches:
- The "Sound-Looker" (AAM): They realized humans look at things that talk or make noise. Their AI combined what it saw with what it heard.
- The "Painter" (SHU-MIIPLab): They used a "diffusion" model (the same tech that generates AI art). They started with a blurry, noisy guess and slowly "cleaned" it up, like a painter refining a sketch, until the perfect attention map appeared.
- The "Dual-Lens" Team (NTR): They used two cameras: one focused on motion (movement) and one focused on stillness (details), then combined the views.

4. Why Does This Matter?

You might ask, "Why do we need a computer to know where I'm looking?"

Smarter Streaming: Imagine a video call. The computer knows you are looking at the person's face, not the background. It can send high-quality video of the face and low-quality video of the background, saving you data and money.
Better Compression: Streaming services can compress videos more efficiently by focusing quality on the "interesting" parts.
Accessibility: It can help robots understand what a human is interested in, making them better assistants.
Editing: It can help editors automatically crop videos to keep the most important parts in the frame.

The Bottom Line

This paper shows that we are getting very good at teaching computers to understand human attention. By using massive amounts of data collected from regular people and combining different smart AI strategies, the winners proved that the future of video isn't just about seeing pixels—it's about understanding what matters to us.

The best part? All the data and the winning code are now free for anyone to use, so the next generation of "super-eyes" can be built on top of this foundation.

1. Problem Definition

The paper addresses the challenge of Video Saliency Prediction, which aims to automatically generate a saliency map for every frame of a video sequence. This map represents the probability density of where a human observer will direct their attention (fixations).

Goal: To model the Human Visual System (HVS) behavior in dynamic environments, capturing both spatial (where) and temporal (when) attention patterns.
Applications: The technology is critical for perceptually guided video compression, objective quality evaluation, adaptive content retargeting, media enhancement, and 3D/VR content optimization.
Challenge: Traditional eye-tracking data is expensive and difficult to scale. The challenge seeks to leverage crowdsourced data to create large-scale, high-quality datasets and develop deep learning models that can predict human gaze accurately without requiring expensive hardware during inference.

2. Methodology and Dataset

2.1. The NTIRE 2026 Dataset

The organizers created a novel, large-scale dataset to overcome the limitations of previous datasets (e.g., Hollywood-2, DHF1K).

Scale: 2,000 diverse videos (1,200 for training, 800 for testing) totaling over 1 million frames.
Source: Curated from YouTube under open CC-BY licenses, covering diverse content types.
Preprocessing: Videos were standardized to FullHD (1920x1080), 30 FPS, and normalized audio (EBU R128).
Annotation: Instead of eye-trackers, the dataset uses crowdsourced mouse tracking from over 5,000 assessors.
- Quality Control: The methodology involved rigorous participant filtering, verification questions, and post-processing (Gaussian smoothing) to ensure the mouse-tracking data closely approximated eye-tracking fixation maps.
- Density: Each video was viewed by an average of >70 assessors.

2.2. Evaluation Metrics

Performance was evaluated on a private test set of 500 videos using four standard metrics:

Pearson's Correlation Coefficient (CC): Measures linear correlation between predicted and ground truth maps.
Similarity (SIM): Intersection over union of the probability distributions.
AUC-Judd: Area Under the Curve for fixation prediction.
Normalized Scanpath Saliency (NSS): Measures the z-score of saliency values at fixation points.

Ranking: Teams were ranked by the average rank across all four metrics.

3. Key Contributions and Winning Methods

The challenge attracted over 20 teams, with 7 teams passing the final code review. The top solutions shared a trend of utilizing large-scale pre-trained video foundation models and multi-scale spatiotemporal modeling.

1st Place: iLearn Team

Core Architecture: A multi-expert ensemble framework built on the InternVideo2 (6B parameter) backbone.
Method:
- Shared Backbone: Extracts multi-level features (layers 11, 23, 35, 47).
- Expert 1 (Temporal-Modulated): Uses a top-down approach with multiplicative gating and explicit spatial priors (center bias) to handle stable temporal dynamics.
- Expert 2 (Multi-Scale): Uses concatenation-based fusion and deep auxiliary supervision (independent heads at intermediate levels) for adaptability to complex, off-center scenes.
- Ensemble: Fuses predictions in logit space (inverse sigmoid $\to$ average $\to$ sigmoid).
Training: Two-stage fine-tuning (frozen backbone first, then LoRA injection).

2nd Place: CVSP Team

Core Architecture: PredJSal, repurposing the V-JEPA2 (Vision Joint Embedding Predictive Architecture) backbone.
Method:
- Theory: Based on Predictive Coding Theory, assuming saliency correlates with prediction errors in a generative model of the world.
- Mechanism: Uses V-JEPA2's self-supervised representations (trained to predict masked video regions) to capture "intuitive physics" and object permanence.
- Decoder: A 3D convolutional decoder with multi-scale skip connections and a lightweight temporal self-attention module to refine features before generating the saliency map.

3rd Place: ARK MMLAB

Core Architecture: Hierarchical model based on InternVideo2-1B.
Method:
- Feature Upsampling: Aligns resolutions of features from different depths.
- Temporal Attention: Computes weights for the deepest features to emphasize salient frames.
- Hierarchical Decoder: Fuses temporally weighted features in a top-down manner, concatenating with shallower branches to recover spatial details.

Other Notable Methods:

Vertex (4th): Extended the TMFI (Transformer-based Multi-scale Feature Integration) network by adding a bottom-up aggregation path to the original top-down fusion, creating a bidirectional design that expands feature scales from 4 to 8.
AAM (5th): Proposed the Attend to Anything Model, a multi-modal approach fusing visual (DINOv3), text (CLIP), and audio (Wav2CLIP) inputs. It uses Hyperbolic Entailment Learning to enforce hierarchical semantic specialization and Fokker-Planck Dynamics to model attention evolution over time.
SHU-MIIPLab (6th): Introduced SSF-DiffNet, a State Space Fusion-driven Diffusion Network. It uses a Mamba-based architecture (SI-MB and SA-MB blocks) to align RGB and optical flow features, guiding a denoising diffusion process to generate saliency maps.
NTR (7th): A Dual-Stream Network combining a video stream (R(2+1)D-18 for motion) and a frame stream (ConvNeXt-Tiny for spatial details), fused with a learnable center prior.

4. Results

Top Performance: The iLearn team achieved the highest average rank, with a CC of 0.8280 and NSS of 3.3229 on the private test set.
Baseline: The organizers' "Center Prior" baseline (a simple Gaussian centered on the image) scored significantly lower (CC ~0.41), highlighting the complexity of the task and the effectiveness of the deep learning approaches.
Trends:
- Foundation Models: All top-performing teams utilized massive pre-trained video backbones (InternVideo2, V-JEPA2) rather than training from scratch.
- Multi-Scale Fusion: Successful methods effectively combined low-level spatial details with high-level semantic abstractions.
- Temporal Modeling: Explicit modeling of temporal dynamics (via attention, gating, or predictive coding) was crucial for distinguishing static background from moving salient objects.

5. Significance and Future Directions

Dataset Advancement: The release of a 2,000-video dataset with high-quality crowdsourced annotations (5,000+ assessors) sets a new benchmark for video saliency research, addressing the scarcity of large-scale data.
Methodological Shift: The results demonstrate a paradigm shift from traditional low-level feature engineering to foundation model-based approaches. The success of V-JEPA2 suggests that self-supervised predictive coding is a powerful inductive bias for saliency.
Multimodality: The inclusion of audio and text in top solutions (AAM) indicates that future saliency models must be inherently multimodal to fully capture human attention.
Open Science: All data, code, and results are publicly available, fostering reproducibility and further research in video understanding, compression, and content generation.

In conclusion, the NTIRE 2026 challenge successfully advanced the state-of-the-art in video saliency prediction by establishing a rigorous benchmark and demonstrating that leveraging large-scale pre-trained video representations yields superior performance in modeling human visual attention.