💻 computer science

QCA: Query- and Content-Aware Keyframe Selection for Long Video Understanding

The paper proposes QCA, a training-free, query- and content-aware keyframe selection framework that dynamically allocates frame budgets and optimizes diversity to achieve state-of-the-art long video understanding performance with significantly fewer frames than existing methods.

Original authors: Jun Peng, Baiyang Song, Jie Li, Hui Li, Yiyi Zhou, Rongrong Ji, Yonghong Tian

Published 2026-07-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jun Peng, Baiyang Song, Jie Li, Hui Li, Yiyi Zhou, Rongrong Ji, Yonghong Tian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to explain a two-hour movie to a friend, but you only have time to show them 128 tiny snapshots (frames) from the film.

If you use the standard method, called "Uniform Sampling," it's like taking a photo every 30 seconds, no matter what's happening. You might catch a shot of a character sleeping, then a shot of them walking, then a shot of them eating. But if the most important moment—the character suddenly pulling a gun—happens at the 29-second mark, your camera snaps a photo of the gun just before it happens and just after it happens, missing the action entirely. You end up with a boring, repetitive slideshow that misses the plot.

This paper introduces a smarter way to pick those snapshots, called QCA (Query- and Content-Aware). Think of QCA as a super-intelligent film editor who watches the whole movie before you ask a question, and then picks the perfect 128 frames to answer your specific question.

Here is how QCA works, broken down into three simple steps:

1. The "Divide and Conquer" Strategy

First, QCA chops the long video into smaller chunks (like chapters in a book). It doesn't treat every minute of the movie the same.

The Analogy: Imagine you are looking for a specific clue in a mystery novel. You wouldn't read every single page with the same intensity. You would skim the boring descriptions of the weather but read the crime scene pages very closely.
What QCA does: It looks at each "chapter" of the video and asks two questions:
1. Relevance: Does this part of the video seem to match the question you asked? (e.g., If you ask "Who is running?", it looks for running scenes).
2. Variety: Is this part of the video different from the rest? (e.g., If the whole movie is a guy sitting in a chair, but suddenly he stands up, that's a "deviation" worth capturing).

2. The "Smart Budget"

Once QCA knows which chapters are important, it decides how many photos to take from each one.

The Analogy: Think of your 128 photos as a budget. If a chapter is boring and irrelevant, QCA spends almost no money on it. If a chapter is the climax of the movie and full of action, QCA spends a huge chunk of the budget there.
The Result: Instead of spreading your photos evenly, you get a pile of photos from the exciting parts and very few from the boring parts.

3. The "Anchor and Expand" Selection

Inside those important chapters, QCA picks the actual frames.

The Anchor: First, it picks the single best frame that answers your question directly. (e.g., The exact moment the gun is pulled).
The Expansion: Then, it looks for other frames in that same chapter that are different from the first one but still relevant.
The Analogy: Imagine you are describing a fight scene. You pick the photo of the punch (the anchor). Then, you look for a photo of the person falling down (diversity). You avoid picking 10 photos of the punch, because that's redundant. You want photos that tell the whole story of that specific moment.

Why is this a big deal?

The paper claims that by using this method, computers (specifically AI models that understand video) can answer questions about long videos much better than before, even when they are forced to look at very few frames.

The Proof: The authors tested this on several difficult video quizzes. When they used QCA with just 128 frames, the AI got a score of 67.8%.
The Comparison: A very powerful, expensive AI (GPT-4o) tried to answer the same questions but was allowed to look at 256 frames (twice as many). Even with double the data, GPT-4o only scored 66.7%.

The Best Part: No Extra Training

Usually, to make an AI smarter, you have to feed it thousands of hours of data to "teach" it how to do the task. This is like hiring a tutor for months.

QCA's Trick: This method requires zero training. It's like giving the AI a new pair of glasses that helps it see better, without changing the AI's brain. You can plug it into almost any existing video-AI system, and it works immediately.

Summary

In short, QCA is a tool that stops video-AIs from wasting time looking at boring, repetitive parts of a movie. Instead, it acts like a smart editor, spending its limited "photo budget" only on the moments that actually matter to your question, ensuring the AI sees the most important evidence to give the right answer.

Technical Summary: QCA - Query- and Content-Aware Keyframe Selection for Long Video Understanding

1. Problem Statement

Long video understanding with Multimodal Large Language Models (MLLMs) is hindered by severe temporal redundancy. Processing dense frame sequences is computationally expensive and often exceeds the token budget limitations of current MLLMs, leading to truncated inputs or degraded reasoning performance. While existing solutions like uniform sampling are efficient, they often overlook semantically critical moments. Conversely, query-based selection methods may capture relevant frames but fail to maintain content diversity, resulting in redundant information. The core challenge is to select a compact, information-rich subset of frames that balances semantic relevance to a specific query with the diversity of video content, all within a strict frame budget, without requiring additional model training.

2. Methodology: QCA Framework

The authors propose QCA (Query- and Content-Aware), a training-free framework designed to integrate seamlessly with existing Video-LLMs. The method operates in two primary stages:

A. Temporal Segmentation

The input video is first divided into $S$ uniformly spaced temporal segments ( $\mathcal{X}_s$ ). This coarse segmentation allows the model to analyze the video structure before selecting specific frames.

B. Inter-Segment Keyframe Allocation

QCA dynamically allocates the total keyframe budget ( $N'$ ) to each segment based on a joint estimation of semantic matching and content deviation:

Semantic Matching ( $M_s$ ): Measures the average relevance of frames in a segment to the query using an Image-Text Matching (ITM) function (e.g., BLIP-2).
Content Deviation ( $D_s$ ): Quantifies the visual distinctiveness of a segment relative to the whole video. It is calculated as the squared Euclidean distance between the segment's mean feature and the video's mean feature, plus the trace of the segment's covariance matrix (capturing intra-segment variation).
Budget Allocation: A contribution score ( $c_s$ ) is computed as a weighted sum of normalized $M_s$ and $D_s$ . A softmax function with a temperature parameter $\tau$ converts these scores into allocation weights, determining the number of frames ( $q_s$ ) assigned to each segment.

C. Intra-Segment Keyframe Selection

Within each allocated segment, QCA selects frames using an anchor-centric greedy strategy:

Semantic Anchor: The frame with the highest ITM score relative to the query is selected first.
Candidate Filtering: A candidate set is constructed by filtering frames that maintain a relevance score above a threshold ( $\gamma$ ) of the anchor's score.
Diversity Maximization: Additional frames are iteratively selected from the candidate set to maximize the aggregate distance (e.g., Euclidean distance in feature space) to the currently selected keyframes. This ensures the selected subset is diverse while remaining semantically anchored to the query.

3. Key Contributions

Formulation: The paper formulates query-conditioned keyframe selection as a joint relevance–diversity modeling and frame allocation problem under a limited budget.
Framework Design: It introduces QCA, a novel framework that combines segment-level contribution estimation (balancing query relevance and content deviation), adaptive budget allocation, and anchor-centric greedy selection.
Training-Free Integration: The method requires no additional training or fine-tuning of the underlying Video-LLM, making it a plug-and-play solution.
Generalization: The approach is demonstrated to be model-agnostic, showing consistent improvements across different Video-LLM backbones (LLaVA-Video, InternVL, Qwen3-VL) and various embedding models (BLIP, CLIP, LongCLIP).

4. Experimental Results

Extensive experiments were conducted on four long video understanding benchmarks: LongVideoBench, Video-MME, MLVU, and LVBench.

Performance Gains: QCA consistently outperforms uniform sampling and existing keyframe selection baselines (e.g., Top-k, AKS, Q-Frame).
- On LongVideoBench using Qwen3-VL-8B with a 64-frame budget, QCA achieves 66.9%, a 3.8% improvement over uniform sampling.
- On Video-MME, it achieves 69.5% (+1.9%), and on MLVU, 75.7% (+4.7%).
- On LVBench, it shows a significant 8.0% gain, reaching 51.8%.
Comparison with SOTA: Using Qwen3-VL-30B-A3B-Instruct, QCA outperforms proprietary models like GPT-4o (on MLVU) and Gemini-1.5-Pro (on LVBench) despite using fewer frames (64 vs. 256 for GPT-4o).
Ablation Studies:
- Removing either semantic matching or content deviation leads to performance drops, confirming the necessity of both components.
- The anchor-centric strategy and candidate filtering are crucial; replacing the anchor with a random frame or removing the candidate set significantly degrades performance.
- Diversity modeling is essential; a simple top-matching strategy (ignoring diversity) performs worse than the full QCA approach.
Efficiency: The preprocessing overhead is minimal. For example, increasing input frames from 128 to 1024 only increases ITM matching time from ~0.6s to ~1.2s, while the selection process itself takes ~0.005s.

5. Significance and Claims

The paper claims that QCA offers a simple yet effective approach to long video understanding that addresses the trade-off between computational efficiency and information retention. By dynamically allocating budgets based on both query relevance and content diversity, QCA enables MLLMs to reason over long videos more effectively without modifying model architectures or incurring training costs.

The authors emphasize that the method's strong generalization ability across different backbones and embedding models highlights its robustness. Furthermore, the results suggest that informed frame selection is more critical than simply increasing the number of input frames or relying on larger proprietary models. The paper concludes that while uniform temporal segmentation is used for simplicity, future work could explore content-aware segmentation (e.g., shot boundary detection) to further enhance performance.