VQPP: Video Query Performance Prediction Benchmark

This paper introduces VQPP, the first benchmark for video query performance prediction comprising 56K text queries and 51K videos, which evaluates various predictors and demonstrates their utility in training large language models for query reformulation.

The Gist

Imagine you are a librarian in a massive, chaotic library filled with millions of video clips. A patron walks up and asks, "Show me a video of a happy horse."

In a perfect world, the librarian would instantly pull up the perfect video. But in the real world, sometimes the librarian finds a great match, and sometimes they pull up a video of a sad horse, or a cartoon horse, or nothing at all.

The Problem: How does the librarian know before they start searching if the patron's request is easy to fulfill or a nightmare? If they know a request is "hard," they might try a different strategy, or they might warn the patron, "Hey, this is tricky, let's try rephrasing it."

This paper introduces a new tool called VQPP (Video Query Performance Prediction) to solve exactly that problem.

The Big Idea: The "Crystal Ball" for Search

The authors built a benchmark (a standardized test) to train computers to act as "crystal balls." Instead of actually searching the library first, the computer looks at the question (the query) and predicts: "Will this search be easy or hard?"

If the computer predicts the search will be hard, the system can automatically fix the question before wasting time searching.

How They Built the Test

To train these "crystal balls," the researchers needed a huge library and a way to grade the searches. They used:

1. Two Giant Libraries: They grabbed data from MSR-VTT (10,000 random, wild videos like news and sports) and VATEX (41,000 shorter, more specific clips).
2. Two Search Engines: They used two different "librarians" (AI models named GRAM and VAST) to actually perform the searches.
3. The Scorecard: They ran 56,000 different questions. For each question, they checked: "Did the librarian find the right video in the top 10 results?"

This created a massive dataset where every question has a known "difficulty score."

The Contest: Who is the Best Predictor?

The researchers tested different types of "predictors" (AI models) to see which one could guess the difficulty best. They split them into two teams:

Team 1: The "Pre-Retrieval" Scouts (The Guessers)
These scouts look only at the question text before searching.

• The Old School Scout: Counts words, checks for confusing grammar, or looks up synonyms. (Like a human guessing, "That's a long sentence, it might be hard.")
• The Deep Learner (BERT): A smart AI that reads the meaning of the sentence. It understands that "a fearful animation scene" is vague, while "a movie scene with Morgan Freeman running in armor" is specific.
• The Chatbot (Llama): A giant language model asked to guess the difficulty based on examples.

Team 2: The "Post-Retrieval" Analysts (The Reviewers)
These analysts wait for the search to finish, look at the list of videos returned, and then say, "Wow, these results look messy, so the search was hard."

• They use complex vision models to compare the videos in the results to see if they make sense together.

The Surprise Winner

You might think the "Reviewers" (Team 2) would win because they have the actual results to look at. But they didn't.

The winner was the Deep Learner (BERT) from Team 1.

• Why? In video search, the "results list" is often very noisy. Even if the search engine fails, the list of wrong videos might look random and confusing, making it hard for the analyst to tell why it failed.
• The BERT model, however, realized that the question itself was the problem. It learned that vague questions lead to bad results, even without seeing the results. It's like a chef who knows a recipe will fail just by reading the ingredients list, without even having to cook it.

The Superpower: Fixing the Questions

The coolest part of the paper isn't just predicting difficulty; it's fixing the questions.

The researchers took their winning "crystal ball" (the BERT predictor) and used it as a coach for a language model (an AI that writes text).

1. The AI writes a new version of a user's question.
2. The "Coach" (BERT) scores the new question: "This one is better! It's more likely to find the right video."
3. The AI learns from this score and gets better at writing questions.

The Result: When they tested this, the AI successfully rewrote vague questions like "a fearful animation scene" into specific ones like "a movie scene starring Morgan Freeman and men in armor running." The search engine then found the right video much more often!

The Takeaway

This paper is a big step forward because:

1. It's the first of its kind: No one had ever built a standardized test for predicting video search difficulty before.
2. It's efficient: You don't need to run expensive video searches to know if a question is good; a smart text-reader can tell you.
3. It's practical: It can automatically improve how we talk to video search engines, making them feel much smarter and more helpful.

In short, VQPP teaches computers to listen to your question and say, "I can help you find that, but let's tweak your wording first so we don't waste time!"

Technical Summary

1. Problem Statement

Query Performance Prediction (QPP), also known as query difficulty estimation, is the task of predicting how well a retrieval system will perform for a specific query without access to ground-truth relevance judgments. While QPP is a mature field in text and image retrieval, it remains largely unexplored in Content-Based Video Retrieval (CBVR).

• Gap: Existing research focuses on text or static images. Video retrieval introduces unique challenges, including temporal dynamics, multimodal complexity (audio, visual, text), and high computational costs.
• Need: There is a lack of standardized benchmarks, datasets, and evaluation protocols specifically designed to train and evaluate QPP models for video.

2. Methodology: The VQPP Benchmark

The authors propose VQPP, the first comprehensive benchmark for video query performance prediction.

Dataset Construction

• Sources: The benchmark aggregates data from two major text-to-video datasets: MSR-VTT (10,000 videos, open-domain "in-the-wild") and VATEX (41,250 videos, shorter clips, English captions).
• Scale: It comprises 56,000 text queries and 51,000 videos.
• Retrieval Systems: To ensure generalization, queries are run against two state-of-the-art CBVR models:
  1. GRAM: A model using a Gramian volume objective for fine-grained text-visual alignment.
  2. VAST: A foundation model integrating auxiliary modalities (audio, subtitles) alongside visual frames.
• Ground Truth: Performance is measured using Reciprocal Rank (RR) and Recall@K (where K=1, 5, 10, 20). Since these datasets have a 1-to-1 query-video mapping, RR and Recall@10 serve as the ground-truth difficulty labels.
• Splits: The data is stratified into official Train, Validation, and Test splits for both datasets and both retrieval systems, creating four distinct evaluation scenarios.

Predictor Categories Evaluated

The benchmark evaluates a wide range of predictors:

1. Pre-Retrieval Predictors: Operate before search, using only the query text.
   • Linguistic Baselines: Statistical heuristics (word count, synset counts, POS tags).
   • Fine-tuned BERT: A regression model predicting RR/Recall directly from the query embedding.
   • Few-shot LLMs: Using Llama-3.1-8B with in-context learning (k-NN selected examples).
2. Post-Retrieval Predictors: Operate after search, analyzing the ranked list of retrieved videos.
   • Fine-tuned CLIP/CLIP4Clip: Binary classifiers distinguishing the ground-truth video from distractors in the top-25 results.
   • Correlation CNN: Analyzes the visual coherence (pairwise correlation matrix) of the top-25 retrieved videos to estimate difficulty.

Application: Query Reformulation

To demonstrate practical utility, the authors use the best-performing predictor (Fine-tuned BERT) as a Reward Model to train a Large Language Model (LLM) for Query Reformulation.

• Method: They employ Direct Preference Optimization (DPO). The LLM (Phi-4-mini) generates multiple reformulations for a query. The BERT predictor scores them; the "winning" (higher predicted performance) and "losing" reformulations form pairs to optimize the LLM.

3. Key Contributions

1. First CBVR Benchmark: Introduction of VQPP, providing 56K queries, 51K videos, and pre-computed retrieval results across two datasets and two retrieval systems.
2. Comprehensive Evaluation: Extensive testing of linguistic, deep pre-retrieval, and post-retrieval predictors across four distinct scenarios.
3. Application Demonstration: Successfully utilizing a QPP predictor as a reward signal to train an LLM for query reformulation, proving the benchmark's utility beyond passive estimation.
4. Open Source: Release of the benchmark, code, and official splits to foster reproducibility.

4. Results and Findings

The experiments yielded several counter-intuitive and significant findings:

• Pre-Retrieval Dominance: Contrary to trends in image retrieval (where post-retrieval often wins), Fine-tuned BERT (Pre-Retrieval) achieved the best performance across all scenarios.
  • Performance: It achieved Pearson correlations up to 0.41 and Kendall correlations up to 0.34.
  • Reasoning: The "weak signal" in video retrieval (only one correct video per query) makes it difficult for post-retrieval models to distinguish difficulty based on the ranked list. Pre-retrieval models rely on the semantic richness of the query itself.
• Dataset Disparity: Performance was significantly lower on VATEX compared to MSR-VTT. The authors hypothesize this is due to VATEX having more concise, less descriptive queries, making difficulty harder to predict.
• Model Invariance: Pre-retrieval predictors showed consistent performance across GRAM and VAST, as they do not depend on the specific retrieval system's output.
• Post-Retrieval Limitations: Even advanced models like Correlation CNN and Fine-tuned CLIP4Clip underperformed compared to the simple BERT regressor.
• Reformulation Success: Using the BERT predictor to train Phi-4-mini via DPO improved retrieval performance.
  • Metric: Recall@10 on MSR-VTT increased from 47.28% (original) to 47.62% (rephrased).
  • Observation: The LLM learned to make queries more descriptive and visually concrete.

5. Significance

• Paradigm Shift: The paper challenges the assumption that post-retrieval analysis is always superior for QPP, showing that in the video domain, deep semantic understanding of the query (pre-retrieval) is more effective.
• Foundation for Future Research: By providing a standardized benchmark, VQPP enables the community to develop specialized predictors for the complex, multimodal nature of video search.
• Practical Impact: The successful application in query reformulation demonstrates that QPP can be used as a real-time feedback loop to improve user search experiences, reducing the need for manual query refinement.
• Scalability: The benchmark's scale (56K queries) is orders of magnitude larger than typical text QPP evaluations, allowing for robust training of deep learning models.

In conclusion, VQPP establishes a rigorous foundation for studying query difficulty in video retrieval, revealing that deep pre-retrieval models currently outperform complex post-retrieval methods and offering a viable pathway for AI-driven query optimization.