💻 computer science

LoVR: A Benchmark for Long Video Retrieval in Multimodal Contexts

This paper introduces LoVR, a large-scale benchmark for long video-text retrieval featuring over 40,000 fine-grained clips and high-quality captions generated via a novel VLM-based refinement pipeline, designed to overcome the limitations of existing datasets and rigorously evaluate advanced multimodal retrieval models.

Original authors: Qifeng Cai, Hao Liang, Zhaoyang Han, Hejun Dong, Meiyi Qiang, Ruichuan An, Quanqing Xu, Bin Cui, Wentao Zhang

Published 2026-02-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qifeng Cai, Hao Liang, Zhaoyang Han, Hejun Dong, Meiyi Qiang, Ruichuan An, Quanqing Xu, Bin Cui, Wentao Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific, tiny moment inside a three-hour movie. You know the scene: a character is wearing a blue shirt, holding a coffee cup, and laughing at a joke. But the movie is so long, and there are so many scenes, that finding that exact second is like looking for a needle in a haystack the size of a city.

This is the problem the paper LoVR is trying to solve.

The Problem: The "Short Video" Trap

Right now, most computer programs designed to search videos are like students who have only ever studied short, 15-second clips. They are great at finding a cat jumping in a 10-second video, but if you give them a 2-hour documentary, they get lost. They don't know how to navigate the long story, or they get confused by the sheer amount of information.

Existing "tests" (benchmarks) for these programs are like using a map of a single room to navigate a whole country. They are too short, the descriptions are too vague, and they don't test the computer's ability to handle long, complex stories.

The Solution: Introducing LoVR

The authors created LoVR (Long Video Retrieval), which is essentially a massive, difficult "final exam" for video-searching computers.

The Library: Instead of short clips, LoVR contains 467 long videos (some over an hour long).
The Details: Inside these long videos, they chopped them up into 40,804 tiny, specific clips.
The Descriptions: For every single clip and the whole video, they wrote very detailed, high-quality descriptions (captions).

Think of it like this: If other tests are like asking, "Find the video with a dog," LoVR asks, "Find the exact 10-second clip where the dog in the red sweater barks at a squirrel while it's raining."

How They Built It: The "Robot Editor" Pipeline

Writing descriptions for 40,000 clips by hand would take humans years. Writing them with a simple robot would result in nonsense. So, the authors built a clever "Robot Editor" pipeline:

The First Draft (The Robot): They used a super-smart AI (a Vision-Language Model) to watch the clips and write the first draft of the description.
The Quality Check (The Grader): Another AI acted as a strict teacher, grading the description. If the description didn't match the video well enough, it was sent back.
The Rewrite (The Editor): If the grade was too low, the system tried again with a different, even smarter AI model.
The Human Touch (The Final Proofreader): Only if the robots failed three times did a human step in to write the description.

This mix of robots and humans allowed them to create a huge dataset that is both massive in size and incredibly accurate.

The "Full Story" Challenge

One tricky part of long videos is that if you just paste all the tiny descriptions together, it reads like a broken sentence. To fix this, the authors taught the AI to act like a novelist. Instead of just listing events, the AI learned to blend the descriptions of the clips together, smoothing out the transitions so the final description of the whole video reads like a coherent story, not a list of bullet points.

The Results: A Reality Check

When the researchers tested the best video-searching computers available today on this new LoVR exam, the results were sobering:

The Struggle: Even the smartest models got lost. They could find the general "video" sometimes, but finding the specific "clip" was very hard.
The Limit: It turns out that just feeding the computer more frames (more pictures per second) doesn't help much. It's not about seeing more; it's about understanding the long story.
The Gap: The best models were still far from perfect, showing that we are a long way from having a computer that can truly "watch" and "understand" a long movie the way a human does.

The Takeaway

LoVR is a new, tougher standard. It's like upgrading a driving test from a parking lot to a busy highway during rush hour. It shows us exactly where current technology is failing and gives researchers a clear target to aim for: building systems that can truly understand long, complex video stories, not just short snippets.

Technical Summary: LoVR - A Benchmark for Long Video Retrieval in Multimodal Contexts

Problem Statement

Video-text retrieval is a critical task in multimodal learning and web-scale search. However, existing benchmarks (e.g., MSR-VTT, DiDeMo, MSVD) are primarily designed for short video clips (average duration < 1 minute) and suffer from three key limitations when applied to real-world long-form content:

Limited Video Length: Current datasets fail to simulate the complexity of long videos (minutes to hours), such as lectures, documentaries, and live streams.
Low Caption Quality: Annotations often rely on machine-generated text (e.g., ASR) which lacks semantic fidelity, or manual annotation which is not scalable and often lacks fine-grained temporal specificity.
Lack of Clip-Level Benchmarks: Existing benchmarks focus on full-video retrieval or pre-segmented short clips that do not originate from continuous long videos, failing to evaluate fine-grained temporal localization within dense temporal contexts.

Consequently, there is an urgent need for a dedicated benchmark that addresses long-form video retrieval and fine-grained clip-level localization simultaneously.

Methodology

Dataset Construction (LoVR)

The LoVR benchmark is constructed from the LongVideoBench dataset, filtered to include only videos exceeding 15 minutes in length. The construction pipeline involves three main stages:

Clip Segmentation:
- Long videos are segmented into temporally contiguous clips using visual content change detection (PySceneDetect).
- A threshold ( $\tau=34$ ) is applied to detect scene changes. The pipeline specifically retains high-scene-dynamic clips while preserving low-scene-dynamic segments to ensure a mix of visual complexity.
- Videos yielding fewer than 40 clips are filtered out to ensure sufficient visual dynamics.
- Result: 467 long videos and 40,804 fine-grained clips.
Caption Generation Pipeline:
To overcome the trade-off between annotation quality and scalability, the authors propose a hybrid "VLM + Human" pipeline:
- Automatic Captioning: Initial captions are generated using the Qwen2.5-VL-7B-Instruct model, prompted to describe events, scenes, character actions, and visual style.
- Quality Assessment: The EVQAScore metric is used to compute a matching score ( $s_i$ ) between the clip and the generated caption. An empirical threshold ( $\theta=0.2$ ) is established based on manual verification of a sampled subset; scores below this indicate low quality.
- Dynamic Optimization: Low-quality captions are regenerated using stronger VLMs (Qwen2.5-VL-72B-Instruct, Qwen2-VL-72B-Instruct) with adjusted temperature settings. This iterative process continues up to three times. Only if a caption fails to meet the threshold after three attempts is manual annotation invoked.
- Full Video Captioning: To generate coherent full-video captions without exceeding LLM context limits, adjacent clip captions are merged. The method aligns the last paragraph of one clip with the first paragraph of the next using an LLM for semantic fusion, rather than simple concatenation.
Final Quality Assessment:
A human evaluation study involving 25 participants rated 300 clips and 100 long videos. The average relevance score was 4.3/5, with over 78% of samples scoring 4 or 5, validating the pipeline's effectiveness.

Experimental Setup

The benchmark evaluates models on two tasks: Text-to-Video/Clip Retrieval and Video/Clip-to-Text Retrieval.

Metrics: Recall@K (R@1, R@5, R@10) and Retrieval Time (with and without I/O).
Baselines: The study evaluates three categories of models:
- Image Encoders: CLIP, MetaCLIP, SigLIP variants, EVA02-CLIP, PE-Core.
- Video Encoders: VideoCLIP-XL, VideoCLIP-XL-v2.
- Multimodal Embedding Models: GME-Qwen2-VL, MM-Embed, LanguageBind-Video.
Variables: Experiments analyze the impact of frame sampling counts (10, 20, 30 frames) on retrieval performance.

Key Contributions

LoVR Benchmark: The introduction of a novel, large-scale dataset specifically for long video-text retrieval, comprising 467 long videos and 40,804 fine-grained clips with high-quality, multi-level captions (clip-level and full-video level).
Scalable Annotation Pipeline: A cost-effective framework integrating VLMs with automated quality scoring (EVQAScore) and iterative refinement, significantly reducing human labor while maintaining high annotation fidelity.
Semantic Fusion for Long Videos: A method to generate coherent full-video captions by semantically fusing clip-level descriptions, preserving contextual information without losing detail.
Comprehensive Evaluation: Systematic experiments revealing the limitations of current state-of-the-art models in long-form scenarios, establishing a new standard for evaluating temporal reasoning and cross-modal alignment.

Results

Extensive experiments on LoVR reveal significant challenges for existing retrieval models:

Performance Gaps: While LanguageBind-Video emerged as the strongest baseline (42.6% R@1 for Text-to-Video, 37.5% R@1 for Video-to-Text), most models struggle with long-form content.
Image Encoder Limitations: Pure image encoders (e.g., CLIP) show poor temporal modeling, achieving only ~23% R@1 in text-to-video retrieval.
Video Encoder Performance: Dedicated video encoders like VideoClip-XL-v2 perform well on clip-level retrieval (55.3% R@1) but show moderate performance on full-video retrieval (~30% R@1).
Multimodal Embeddings: Models like GME-Qwen2-VL and MM-Embed excel in clip-level tasks but suffer substantial performance drops on long videos, indicating difficulty in generalizing image-trained embeddings to extended temporal contexts.
Frame Sampling Impact: Increasing the number of sampled frames (from 10 to 30) yields marginal or inconsistent improvements. In some cases (e.g., Text-to-Clip), fewer frames performed better, suggesting that naive frame sampling introduces noise rather than useful signal.
Benchmark Difficulty: Even the best-performing models fail to achieve perfect recall at large K, highlighting the intrinsic difficulty of long-range temporal reasoning and semantic grounding required by LoVR.

Significance

The paper positions LoVR as a necessary evolution in video-text retrieval research. By explicitly targeting long-form content, LoVR surfaces technical bottlenecks in current approaches, particularly regarding computational efficiency, sparse semantic signals in long durations, and the alignment of rich, long captions with visual content. The authors claim that LoVR provides a rigorous and realistic testbed that will drive the development of more robust retrieval systems capable of handling the complexities of real-world web-scale video search. The dataset and code are released to facilitate future research in this domain.