💻 computer science

HiVid: LLM-Guided Video Saliency For Content-Aware VOD And Live Streaming

HiVid is a novel framework that leverages Large Language Models as a scalable human proxy to generate high-fidelity, content-aware video saliency weights for both VOD and live streaming, overcoming modality and latency challenges through specialized perception, ranking, and prediction modules to significantly improve subjective quality of experience.

Original authors: Jiahui Chen, Bo Peng, Lianchen Jia, Zeyu Zhang, Tianchi Huang, Lifeng Sun

Published 2026-02-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiahui Chen, Bo Peng, Lianchen Jia, Zeyu Zhang, Tianchi Huang, Lifeng Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive video streaming service (like Netflix or YouTube Live). Your goal is to make sure everyone has a smooth, high-quality viewing experience, even when their internet connection is shaky.

To do this, your system uses a smart algorithm called ABR (Adaptive Bitrate). Think of this algorithm as a traffic cop for data. It decides how much "data fuel" to send to your screen. If the road is clear, it sends high-definition fuel. If the road is jammed, it switches to a lower quality to prevent the video from freezing (buffering).

The Problem:
Traditionally, this traffic cop treats every second of the video the same. It doesn't know if you are watching a boring lecture or an exciting movie climax.

The Old Way: It gives the same amount of data to a boring scene as it does to an explosion. This is wasteful.
The Human Way: You could hire thousands of people to watch every video and say, "This part is boring, give it low data. This part is amazing, give it high data!" But this is too expensive and too slow.
The AI Way: You could use computer vision AI, but they are often "dumb" about context. They might think a static shot of a face is boring, even if it's the most emotional moment in the movie.

The Solution: HiVid
The authors of this paper built HiVid, a new system that acts like a super-smart, tireless film critic powered by a Large Language Model (LLM) (the same tech behind chatbots like me).

Here is how HiVid works, broken down into three simple steps using a Concert Analogy:

1. The Perception Module: The "Zoom-In" Critic

The Challenge: An LLM can't read a whole 2-hour movie in one go (it has a memory limit). If you try to feed it the whole thing, it gets confused or runs out of memory.
The HiVid Fix: Instead of asking the critic to watch the whole concert at once, HiVid breaks the concert into small 5-minute chunks.

It shows the critic a few key frames from the current chunk.
The critic says, "Okay, this part is a bit slow, but the singer is building up energy."
Crucially, the critic keeps a running diary (a summary) of what happened in the previous chunks. So, when it sees the singer start a solo, it remembers, "Ah, this is the climax we were waiting for!"
Result: The system gives a "hype score" to every chunk of the video, knowing exactly where the exciting parts are.

2. The Ranking Module: The "Global Editor" (For Recorded Videos)

The Challenge: If you rate the concert in 5-minute chunks separately, the critic might be inconsistent. They might give a 9/10 to a quiet moment in the first chunk and a 6/10 to a louder moment in the last chunk, just because they forgot the big picture.
The HiVid Fix: Once the critic has rated all the chunks, HiVid acts like a Global Editor.

It takes all the scores and re-organizes them.
It uses a special "Merge Sort" algorithm (a way of sorting lists) where the LLM acts as the judge to compare two chunks at a time. "Is this explosion scene more important than that dialogue scene?"
Result: The final list of scores is perfectly consistent. The most exciting moments get the highest priority, and the boring parts get the lowest, creating a smooth, logical map of the video's importance.

3. The Prediction Module: The "Fortune Teller" (For Live Streams)

The Challenge: In a live stream, the video is happening right now. You can't wait for the critic to watch the next 5 minutes before you decide what data to send. You need to know what's coming before it happens. Also, the critic is slow (it takes time to think).
The HiVid Fix: HiVid uses a Time-Traveling Crystal Ball.

While the critic is slowly rating the current chunk, a separate "Forecasting Model" looks at the pattern of the past ratings.
It predicts: "Based on the rhythm of the music so far, the next 30 seconds will be a high-energy dance break."
It adjusts its prediction speed to match how fast the critic is working. If the critic is slow, the crystal ball predicts further into the future to fill the gap.
Result: The traffic cop (ABR) gets the data it needs before the video even plays, ensuring the live stream never buffers, even during the most intense moments.

Why This Matters

For You (The Viewer): You get better quality during the exciting parts and fewer freezes during the boring parts. Your "Quality of Experience" (QoE) goes up.
For the Streamer: They save money on bandwidth because they aren't wasting data on boring scenes.
The Magic: HiVid proved in tests that it is 11.5% more accurate than the best existing AI for recorded videos and 26% better for live streams. It bridges the gap between the cheap speed of AI and the expensive accuracy of human judgment.

In a nutshell: HiVid is a system that uses a smart AI critic to read a video's "emotional map," organizes that map so it makes sense, and predicts the future excitement levels to ensure your video stream is always smooth and crisp.

1. Problem Statement

Content-aware video streaming aims to optimize the Quality of Experience (QoE) by allocating higher bitrates to video chunks that are more important to the user (saliency). The QoE model is defined as $QoE = \sum w_i \times q_i$ , where $w_i$ is the content importance weight and $q_i$ is the objective quality.

The core challenge lies in generating accurate, scalable, and low-latency saliency weights ( $w_i$ ):

Human Annotation: While accurate (e.g., SENSEI), it is prohibitively expensive and time-consuming, making it unscalable for large-scale Video-on-Demand (VOD) and impossible for live streaming.
Traditional Computer Vision (CV) Models: Models like DETR or VASNet generalize poorly to diverse video categories and lack deep semantic understanding, resulting in low correlation with human preferences.
Large Video Models: Models like VideoLLaMA3 or VILA excel at captioning but often fail at zero-shot subjective rating tasks, producing hallucinated or inconsistent scores.
LLM Constraints: While Large Language Models (LLMs) possess strong reasoning capabilities, they lack native video modalities and have strict token limits (e.g., 128k), preventing them from processing entire long videos or live streams in a single pass.

2. Methodology: The HiVid Framework

HiVid is the first framework to leverage LLMs as a scalable "human proxy" for generating high-fidelity saliency weights. It addresses three specific challenges through a three-module pipeline:

A. Perception Module (Addressing Modality & Context Limits)

Goal: Assess video chunks without exceeding LLM token limits or requiring full video input.
Mechanism:
- Sliding Window: The video is processed in local windows of $m$ frames (e.g., 10 frames).
- Anchor Sampling: Only anchor frames (e.g., the first frame of each chunk) are sampled to reduce redundancy.
- Iterative Context: The LLM rates the current window based on the sampled frames and a periodical video summary generated from previous windows.
- Output: This generates local group ratings and updates the global video summary, allowing the system to handle arbitrary video lengths with minimal overhead.

B. Ranking Module (Addressing VOD Rating Inconsistency)

Goal: Eliminate rating discrepancies caused by the lack of global context in local windows (e.g., a "climax" in one window might be rated lower than a "celebration" in another if the LLM lacks the full story).
Mechanism:
- LLM-Guided Merge Sort: A novel algorithm that uses the LLM as the comparison function to sort and merge groups of frames.
- Global Re-ranking: The module takes the local ratings and the global video summary to perform a global re-ranking of all chunks.
- Gaussian Smoothing: The final discrete rankings are normalized and smoothed to match the continuous nature of human perception, reducing oscillation in the weights.

C. Prediction Module (Addressing Live Streaming Latency)

Goal: Predict future chunk weights in real-time without access to future video content, compensating for variable LLM inference latency.
Mechanism:
- Asynchronous Pipeline: While the LLM processes the current window, a forecasting model runs in parallel to predict future weights.
- Multi-Modal Time Series Model: The predictor takes three inputs:
  1. Historical time-series weights.
  2. Visual features (via CLIP).
  3. Textual summaries (via CLIP).
- Content-Aware Attention: A novel attention mechanism where the time-series features act as the Query, and the concatenated visual/text features act as Key/Value. This allows the model to learn how specific content contexts influence the evolution of saliency scores.
- Adaptive Horizon: The prediction output dimension ( $L_{out}$ ) is dynamically adjusted based on the LLM latency ( $\Delta t$ ) and forecasting time ( $\delta$ ) to ensure the ABR algorithm always has weights for the upcoming chunks.

3. Key Contributions

First LLM-Guided Pipeline: HiVid is the first systematic framework to use LLMs as a scalable proxy for content-aware streaming, bridging the gap between expensive human annotation and inaccurate CV models.
Three-Tiered Architecture:
- Perception: Solves token limits via sliding windows and iterative summarization.
- Ranking: Solves local-global inconsistency via an LLM-guided merge-sort algorithm.
- Prediction: Solves live streaming latency via a multi-modal, content-aware time-series forecasting model with adaptive horizons.
Novel Loss Function: Introduces a correlation-based loss function for the forecasting model to better capture relative preference distributions rather than just absolute error.

4. Experimental Results

The framework was evaluated on three datasets (YouTube-8M, TVSum, SumMe) and validated via a real-world user study.

VOD Performance (Saliency Accuracy):
- HiVid outperformed 8 State-of-the-Art (SOTA) baselines (including DETR, VideoLLaMA3, VILA, and MLPs).
- Improvements: +11.5% in Pearson Linear Correlation Coefficient (PLCC), +6% in Mean Average Precision (mAP), and +14.7% in Mean Opinion Score (MOS) correlation compared to the best baseline.
- Cost-Efficiency: While human annotation costs ~$100/video, HiVid costs ~$1.35/video (using GPT-4o) and ~$0.44 (using GPT-4o-mini).
Live Streaming Performance (Forecasting):
- HiVid outperformed 9 SOTA forecasting baselines (including iTransformer, TimeMixer, and LLM-based prompts).
- Improvements: +26% improvement in forecasting accuracy (PLCC) while guaranteeing real-time latency.
- Latency: The asynchronous design imposes near-zero latency on the underlying ABR algorithm.
User Study:
- A study with 10 volunteers on 320 video clips showed that HiVid's weighted QoE model correlated significantly better with human ratings (MOS) than existing methods, validating its ability to capture subjective importance.
Robustness:
- The framework demonstrated stability across different LLM backbones (GPT-4o, Llama-3, Qwen, etc.) and sensitive categories (Politics, People), with low standard deviation in ratings.

5. Significance

HiVid represents a paradigm shift in adaptive bitrate (ABR) streaming. By effectively leveraging the reasoning capabilities of LLMs while mitigating their limitations (modality, token limits, latency) through specialized architectural modules, it enables content-aware streaming at scale.

For VOD: It allows for dynamic, high-quality streaming that prioritizes user interest without the cost of crowdsourcing.
For Live Streaming: It provides the first viable solution for real-time, content-aware bitrate optimization, ensuring that critical moments in live events (e.g., sports goals, news breaking) receive higher bandwidth, thereby significantly improving the viewer's subjective experience.

The paper establishes that combining LLMs with traditional time-series forecasting and video understanding creates a robust system that outperforms both pure vision models and pure human annotation in terms of the accuracy-to-cost ratio.