🤖 AI

HAS: Highlight-guided Attention Steering for Multimodal LLM Video Summarization

This paper introduces HAS (Highlight-guided Attention Steering), a novel method for multimodal LLM video summarization that improves coherence and information retention by generating a global continuous frame-level highlight distribution to steer the model's attention toward key moments without completely ignoring less significant frames.

Original authors: Rui Chu, Yingjie Lao

Published 2026-07-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rui Chu, Yingjie Lao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to explain a two-hour movie to a friend who only has five minutes to listen. You have a choice: you can either skip the boring parts and only tell them about the explosions and the big kiss, or you can tell the whole story but emphasize the exciting parts while still mentioning the quiet moments so the plot makes sense. This is the heart of video summarization, a field where computers try to do exactly that.

In the past, computers were like clumsy editors who would cut out entire scenes they thought were unimportant, often leaving the story broken or confusing. But recently, we've built super-smart AI brains called Multimodal Large Language Models (M-LLMs). Think of these as AI that can "watch" a video and "read" it at the same time, understanding both the pictures and the words. The big question researchers are asking is: How do we get these super-smart AIs to summarize a video perfectly without them forgetting the important details or getting distracted by the boring parts? The answer isn't to force the AI to pick and choose frames like a human editor; it's to gently guide its attention.

This is where a new paper from Tufts University comes in, introducing a clever trick called HAS (Highlight-guided Attention Steering). The researchers argue that the old way of summarizing videos—picking specific "key frames" and ignoring the rest—is like trying to understand a song by only listening to the chorus. You miss the verses, the bridge, and the flow. Instead, HAS suggests we should let the AI watch the entire video, but give it a "mental highlighter" that glows brighter over the exciting parts and dimmer over the quiet parts.

Here is how HAS works, using a simple analogy: Imagine the AI is a student taking a test on a long, complex video. Before the test, instead of giving the student a list of specific pages to study (which might make them forget the rest of the book), the teacher gives them a highlighted map. On this map, the most important moments are glowing gold, and the less important moments are just a soft gray. The student still reads every single word (the AI still processes every frame), but because of the glowing map, their eyes naturally linger longer on the gold parts. They remember the gold parts better, but they don't forget the gray parts entirely, so the story stays connected.

In technical terms, the paper proposes a two-step process. First, the system creates a smooth, continuous "highlight distribution" for the video. This isn't a hard list of "important" or "unimportant" frames; it's a gentle curve that says, "This moment is 90% important, this one is 60%, and this one is 20%." Second, this curve is turned into a "steering vector"—a tiny signal that is injected into the AI's brain right while it is generating the summary. This signal acts like a nudge, telling the AI's attention mechanism to focus more energy on the high-scoring moments without completely ignoring the low-scoring ones.

The authors tested this idea on several different video datasets, ranging from short clips to long scientific lectures. They found that HAS consistently outperformed older methods that relied on cutting out frames. For instance, when summarizing long academic talks, HAS was better at keeping the facts straight and ensuring the summary matched the video evidence. The paper suggests that by avoiding the "hard cut" of deleting frames, HAS preserves the context needed to make a coherent story. It's a "plug-and-play" method, meaning it works with many different types of AI brains without needing to retrain them from scratch.

The researchers are careful to note that while HAS is a significant improvement, it's not a magic wand that solves every problem. The quality of the summary still depends on how well the initial "highlight map" is drawn. If the map is wrong, the AI will still get confused. However, the results suggest that this gentle steering approach is a much better way to handle the complexity of video than the old "cut and paste" methods. It allows the AI to be both efficient and thorough, capturing the excitement of the highlights while keeping the full story intact.

In the end, HAS shows us that sometimes the best way to summarize a long story isn't to chop it up, but to guide the listener's attention so they know exactly where to look, ensuring nothing important is lost in the noise.

Technical Summary: HAS (Highlight-guided Attention Steering for Multimodal LLM Video Summarization)

1. Problem Statement

Video summarization aims to generate concise summaries from long video sequences while preserving essential content. Current approaches, particularly those leveraging Multimodal Large Language Models (M-LLMs), often rely on a "select-then-summarize" pipeline. This involves identifying discrete key frames or segments (hard selection) and feeding only these subsets to the model. The authors argue that this approach has three critical limitations:

Loss of Coherence: Discrete selection disrupts the temporal continuity of the video, harming the model's ability to understand the narrative flow.
Information Loss: Discarding non-selected frames results in the loss of potentially important contextual information, even if those frames are not "highlights."
Underutilization of M-LLM Capacity: By forcing a hard selection before inference, the method fails to leverage the M-LLM's inherent attention mechanisms to dynamically weigh frame importance during generation.

The core challenge addressed is how to guide an M-LLM to focus on important moments during inference without discarding the continuous video context or requiring expensive fine-tuning.

2. Methodology: HAS Framework

The authors propose HAS (Highlight-guided Attention Steering), a training-free, inference-time framework that treats video summarization as a continuous process guided by a highlight distribution. The method consists of two primary stages:

A. Continuous Highlight Distribution Construction

Instead of selecting discrete frames, HAS generates a smooth, continuous temporal importance curve for the entire video.

Highlight Generation: An off-the-shelf highlight/moment detection module (e.g., query-conditioned saliency models) processes the input video $F$ and user query $q$ to produce a raw temporal score sequence $\hat{h}$ .
Temporal Calibration: The raw scores are often noisy or clip-based. HAS applies linear interpolation to align the scores with the video's frame length $T$ and normalizes them to the range $[0, 1]$ .
Smoothing: The result is a bounded, continuous highlight distribution $h = [h_1, \dots, h_T]$ , representing the probability of importance for every frame.

B. Attention Steering Injection

The continuous distribution $h$ is converted into a steering vector $V$ and injected into the M-LLM during the decoding process.

Vectorization: The frame-level distribution is lifted to the token level. Since each frame $F_t$ corresponds to $P$ visual tokens, the score $h_t$ is repeated $P$ times. A logarithmic transformation is applied to form the steering vector:
$\mathbf{V} = \log(\text{Repeat}(\mathbf{h}, P) + \epsilon)$
The constant $\epsilon$ ensures numerical stability, preventing low-score frames from collapsing to $-\infty$ and ensuring the method remains a "soft" steering mechanism rather than a hard filter.
Inference-Time Intervention: During the autoregressive decoding of the summary, HAS modifies the pre-softmax attention logits of a selected subset of visual cross-attention heads $(\ell, m)$ .
$\tilde{\mathbf{A}}^{(\ell,m)}_{i,:} = \mathbf{A}^{(\ell,m)}_{i,:} + g_{\ell,m} \beta_{\ell,m} \mathbf{V}$
Here, $g_{\ell,m}$ is a learnable gate (sigmoid of a parameter $a_{\ell,m}$ ) and $\beta_{\ell,m}$ is a steering strength. This additive bias increases the attention weight on high-highlight tokens while maintaining non-zero attention on low-highlight tokens.
Optimization: The steering parameters $\Theta = \{a, \beta\}$ are optimized on a validation set to minimize the negative log-likelihood (NLL) of reference summaries, keeping the M-LLM backbone and the highlight generator frozen. A sparsity term encourages the use of only a small subset of attention heads.

3. Key Contributions

The paper claims the following contributions:

First Application of Attention Steering to Video Summarization: HAS is the first work to utilize inference-time attention steering specifically to guide M-LLMs for video summarization.
Continuous Perspective: Unlike prior methods that rely on discrete frame selection, HAS treats summarization as a continuous process, enhancing the temporal consistency of the generated output.
Training-Free and Lightweight: The framework operates entirely at inference time without requiring fine-tuning of the backbone M-LLM, making it easily adaptable to various pre-trained models.
Performance Gains: Extensive experiments demonstrate improvements in temporal coherence, summarization quality, and scalability across multiple benchmarks.

4. Experimental Results

The authors evaluated HAS on three benchmark families: SumMe and TVSum (V2V), VideoXum (Cross-modal), and VISTA (Scientific long-form).

Temporal Importance Alignment: On SumMe and TVSum, HAS achieved state-of-the-art rank correlations (Kendall's $\tau$ and Spearman's $\rho$ ) with human annotations, outperforming both classical visual summarizers and LLM-centric baselines like LLMVS.
Cross-Modal Consistency: On VideoXum, HAS improved video-to-text (V2T) quality while significantly strengthening video-to-video (V2V) salience and video-text semantic alignment (FCLIP), indicating better grounding of the summary in the video evidence.
Long-Form Groundedness: On the VISTA benchmark (scientific talks), HAS improved factual consistency (FactVC) and video-grounded alignment (VideoScore) without sacrificing overall summary quality (ROUGE-L).
Coverage and Information Retention: Analysis of coverage vs. budget showed that while discrete selection methods saturate early (losing information), HAS continues to improve coverage at mid-to-high budgets, demonstrating reduced information loss.
Scalability: HAS showed consistent performance improvements across six different open-source video MLLM backbones (e.g., mPLUG-Owl3, LLaVA-NeXT, Video-LLaVA) in a zero-shot setting, proving its backbone-agnostic nature.

5. Significance and Claims

The paper positions HAS as a practical solution to the "information loss" problem inherent in current video summarization pipelines. By shifting from hard selection (discarding frames) to soft steering (biasing attention), HAS allows M-LLMs to utilize their full capacity to process continuous video context.

The authors claim that HAS offers a "practical inference-time mechanism" for faithful summarization, particularly in high-stakes domains like scientific presentations where factual completeness is critical. They emphasize that the method does not require expensive retraining, making it a viable tool for adapting existing models to better handle long-form video understanding.

The paper acknowledges limitations, noting that the quality of the steering signal depends on the initial highlight generator and that summaries remain inherently lossy. Future work is proposed to explore stronger prior-generation strategies and combine coarse steering with finer temporal localization mechanisms.