💻 computer science

ViASNet: A Video Ad Saliency Network for Predicting Dynamic Saliency and Viewer Engagement

This paper introduces ViASNet, a deep learning model based on a 3D U-Net architecture that integrates audio and semantic cues to predict dynamic saliency maps for short-form video ads, demonstrating its ability to optimize ad design and identify unengaging content through eye-tracking validation and entropy-based diagnostics.

Original authors: Jianping Ye, Michel Wedel

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jianping Ye, Michel Wedel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a director filming a 30-second commercial. You want to make sure that when people watch it, their eyes land exactly where you want them to: on the product, the smiling actor, or the catchy logo. But here's the problem: you can't ask a million people to watch your ad and track their eyes for every single frame. That would be too expensive and too slow.

This paper introduces ViASNet, a smart computer program designed to solve that problem. Think of ViASNet as a "super-observer" that can watch a video ad and instantly predict exactly where a human's eyes will go, second by second.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Clutter" of Ads

We are drowning in short video ads on TikTok, YouTube, and TV. Because there are so many, it's hard for any single ad to grab your attention. Advertisers need a way to test their ads quickly to see which ones work and which ones are boring. Usually, they have to hire people, hook them up to eye-tracking cameras, and watch them stare at screens. ViASNet aims to do this automatically.

2. The Solution: A "Three-Sense" Detective

Most old computer models for predicting eye movement only looked at the picture (visuals). They were like a detective who only looks at a crime scene but ignores the sound or the story.

ViASNet is different. It is a "three-sense" detective that looks at three things simultaneously:

The Visuals: What is happening on the screen? (Is there a face? Is something moving?)
The Audio: What is being said or heard? (Does a loud noise or a voice make you look up?)
The Story (Semantics): What is the scene about? (Is it a sad moment? A funny joke? A car chase?)

To understand the "story," the system uses a giant AI brain (called Qwen3-VL) to read the video and write a short caption describing the scene, just like a human would.

3. How It Builds the Map: The "Saliency Map"

The goal of ViASNet is to create a Saliency Map. Imagine taking a photo of the video frame and painting it with a heat map:

Red/Hot areas: Where people will look.
Blue/Cold areas: Where people will ignore.

ViASNet builds this map for every single frame of the video, creating a moving heat map that shows exactly how attention shifts over time.

4. The Secret Sauce: How It Learns

The researchers trained ViASNet using data from 151 real TV commercials that were watched by about 20 people each while their eyes were tracked by high-tech cameras.

The model learned by comparing its own predictions against the real eye movements of those 20 people. It uses a special architecture called 3D U-Net.

The Analogy: Think of the U-Net like a funnel. It takes the video, squeezes it down to understand the main ideas (the "bottleneck"), and then expands it back out to draw the detailed heat map.
The "Center Bias": Humans naturally tend to look at the center of the screen. ViASNet knows this trick and adds a little "center bias" to its predictions, just like a real human would.

5. The "Entropy" Test: Measuring Boredom

One of the coolest tools the authors created is a way to measure Entropy (a fancy word for "disorder" or "confusion").

Low Entropy (Orderly): If the heat map shows everyone looking at one spot (like a face in the center), the entropy is low. This means the ad is engaging.
High Entropy (Chaotic): If the heat map is scattered all over the place, or if people are looking at nothing in particular, the entropy is high. This means the ad is confusing or boring.

The authors used this to test 15 new ads they had never seen before. They could instantly spot which scenes were causing viewers to lose focus. For example, they found that if a scene had too many objects or text scattered everywhere, viewers' eyes would wander, and the "engagement score" would drop.

6. The Results: It Works Better Than the Rest

The researchers tested ViASNet against other famous computer models (some that only look at pictures, some that look at movies).

The Verdict: ViASNet won. It predicted where people would look more accurately than any other model they tested.
Why? Because it didn't just look at the pixels; it understood the sound, the story, and the cuts between scenes. It realized that when a scene changes (a "cut"), people's eyes naturally jump to the center to re-orient themselves, and it accounted for that.

Summary

In short, ViASNet is an automated tool that lets advertisers "see" through the eyes of their audience without needing a single human to sit in a lab. It combines what you see, what you hear, and what the story means to predict exactly where your attention will go. If an ad is boring or confusing, this system can tell you exactly which second it happens, allowing creators to fix it before they spend money on a big marketing campaign.

Technical Summary: ViASNet for Video Ad Saliency Prediction

Problem Statement

The digital media landscape has shifted toward short-form video advertising across TV, social media, and e-commerce platforms. While these ads can achieve high engagement, their omnipresence creates significant "clutter," making it difficult for advertisers to design content that captures viewer attention. Traditional methods for assessing ad effectiveness rely on neuro-physiological measurements, particularly eye tracking, to analyze fixation variability. However, collecting and processing eye-tracking data is costly and time-consuming, hindering rapid, large-scale testing of the high volumes of video ads being produced, especially with the rise of generative AI.

Existing computational models for saliency prediction face several challenges when applied to video ads:

Temporal Dynamics: Modeling the simultaneous temporal dynamics of video frames and eye movements is computationally difficult.
Scene Cuts: Video ads feature rapid, purposeful scene changes that disrupt eye movement patterns, requiring models to account for abrupt transitions.
Semantic and Audio Influence: Saliency is driven not only by low-level visual features (contrast, motion) but also by high-level scene semantics and audio streams, which can enhance specific visual features.
Limitations of Current Models: State-of-the-art dynamic saliency models often predict saliency only for the final frame of a clip or fail to incorporate scene narratives and audio effectively.

Methodology: ViASNet

The authors propose ViASNet (Video Ad Saliency Network), a deep learning architecture designed specifically for end-to-end dynamic saliency prediction in short-form video ads. The model is built upon a 3D U-Net structure and integrates multimodal inputs to predict frame-by-frame saliency maps.

Architecture Components

Input Preprocessing:
- Video ads are segmented into scenes using PySceneDetect.
- Visual frames are resized to $224 \times 384$ .
- Raw audio signals are normalized to $[-1, 1]$ and aligned with visual frames, avoiding frequency-domain conversion to reduce preprocessing subjectivity.
- Ground-truth saliency maps are generated from eye-tracking data using Gaussian blurring and min-max normalization.
Encoders:
- Vision Encoder: Utilizes a pre-trained S3D backbone (factorized 3D convolutions) designed for motion recognition. It processes batches of scenes to extract spatial and temporal features.
- Audio Encoder: A customized 7-layer CNN with a feature-pyramid architecture (combining bottom-up and top-down pathways with lateral connections) to process raw audio signals.
- Scene Semantics Encoder: Uses the Large Multimodal Model Qwen3-VL-Plus to generate scene captions. To capture narrative continuity, the caption of the previous scene is injected into the prompt for the current scene. These captions are then encoded using a fine-tuned BERT (BERT-base-uncased) model.
Multimodal Fusion:
- Features from vision, audio, and semantics are concatenated and processed by a Multimodal Unified Attention Network (MUAN).
- This block employs self-attention for intra-modal dependencies and co-attention for inter-modal dependencies (e.g., audio-video, video-semantic), allowing the model to capture complex interactions between modalities.
Decoder and Readout:
- The decoder follows a U-Net structure with transposed convolutions for upsampling, utilizing skip connections to preserve spatial details from the encoder.
- A Readout Network (similar to a ResNet bottleneck block) processes the combined features, reducing channel depth and capturing spatial patterns before restoring the original depth.
Output Refinement:
- The final output incorporates a Center Bias component (modeled via a trainable 2D Gaussian kernel) to account for the tendency of viewers to fixate on the center of the screen.
- A trainable Gaussian Blur is applied to the output to smooth the saliency map, followed by a softmax activation to produce a probability distribution.

The model is trained end-to-end using the Kullback-Leibler (KL) divergence between the predicted and ground-truth saliency maps.

Key Contributions

Novel Architecture: The development of ViASNet, a specialized deep saliency network that integrates visual, audio, and semantic scene information via a unified attention mechanism.
Multimodal Integration: Explicit modeling of the influence of audio streams and scene semantics (via LLM-generated captions) on visual saliency, addressing a gap in previous dynamic saliency models.
Diagnostic Tool: The introduction of frame-by-frame entropy calculated from predicted saliency maps as a metric to diagnose viewer engagement. High entropy indicates low engagement (scattered attention), while low entropy indicates high engagement (focused attention).
Comprehensive Evaluation: Extensive testing on a dataset of 151 video ads (136 for training, 15 for testing) with eye-tracking data from approximately 20 viewers per ad.

Results

The model was evaluated against four baselines: Itti-Koch (computational), BMS (computational), DeepGaze IIE (deep learning, static-trained), and ViNet-S (deep learning, dynamic-trained).

Performance Metrics: ViASNet achieved state-of-the-art performance across all six evaluation metrics (KL, CC, NSS, SIM, AUC, s-AUC) on the test set.
- It outperformed DeepGaze IIE by over 20% in AUC and significantly more in other metrics, highlighting the benefit of dynamic modeling over frame-wise static prediction.
- It surpassed ViNet-S on most metrics, demonstrating the efficacy of its specific architecture for video ads.
Ablation Studies: Removing components confirmed the importance of the Unified Attention mechanism (4.0% increase in KL when removed) and the Readout network (3.8% increase). While audio and caption attention had smaller individual impacts, they contributed measurably to performance.
Entropy Analysis: The study demonstrated that predicted entropy correlates with ground-truth fixation variability (correlation of 0.51). The entropy metric successfully identified specific scenes and entire ads that failed to engage viewers (e.g., scenes with high entropy due to scattered objects/text) and those that maintained attention (low entropy, focused on faces/products).

Significance and Claims

The authors position ViASNet as a practical solution to the challenges of scaling video ad testing in an era of high-volume, low-cost production. The paper claims that:

Automation: ViASNet enables the automated prediction of dynamic saliency maps for unseen video ads, reducing the reliance on expensive and slow eye-tracking studies.
Diagnostic Capability: By calculating frame-by-frame entropy, the system can identify specific scenes or ads that fail to capture attention, allowing advertisers to modify or remove underperforming content before release.
Human-in-the-Loop: The authors modestly suggest that ViASNet should not replace human ad testing but rather serve as a tool in Human-in-the-Loop (HITL) settings. This approach combines the speed of automated testing with human strategic oversight to guide ad design, ensuring authenticity and ethical considerations are maintained.

The study concludes that deep saliency models like ViASNet can significantly accelerate ad design and testing cycles, providing critical diagnostics for optimizing viewer engagement in the competitive short-form video landscape.