💻 computer science

VIRTUE: Versatile Video Retrieval Through Unified Embeddings

VIRTUE is a versatile video retrieval framework that leverages a unified multimodal LLM backbone with contrastive alignment and LoRA training to achieve state-of-the-art zero-shot performance across corpus retrieval, moment localization, and composed multimodal queries, rivaling specialized systems trained on significantly larger datasets.

Original authors: Shaunak Halbe, Bhagyashree Puranik, Jayakrishnan Unnikrishnan, Kushan Thakkar, Vimal Bhat, Toufiq Parag

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shaunak Halbe, Bhagyashree Puranik, Jayakrishnan Unnikrishnan, Kushan Thakkar, Vimal Bhat, Toufiq Parag

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of millions of videos, but instead of books, they are all just floating in a dark room. You want to find a specific scene, like "a dog chasing a ball in the snow," or even a more complex request like, "Show me a video of a beach, but make it look like it's raining."

Currently, finding these videos is like trying to find a needle in a haystack using different tools for different jobs. Some tools are great at finding whole videos based on a text description, but they get confused if you show them a picture and ask them to change the weather. Other tools are great at complex questions but are terrible at actually finding the right video quickly.

Enter VIRTUE (Versatile vIdeo Retrieval Through Unified Embeddings). Think of VIRTUE as a super-smart, all-in-one librarian who can do everything.

The Problem: The "Specialist" vs. The "Generalist"

The Specialists: Imagine a team of librarians where one only knows how to sort books by title, and another only knows how to sort by author. They are very fast and accurate at their specific job, but if you ask the "Title" librarian to sort by "Cover Color," they have no idea what to do. In video tech, these are the specialized models. They are great at finding videos from text, but they can't handle complex, mixed requests (like "this video, but with a different background").
The Generalists: Then you have the "Multimodal Large Language Models" (MLLMs). These are like a librarian who can read, see pictures, and understand complex stories. They are very smart and can handle any request you throw at them. However, they are often slow and not very precise when it comes to actually finding the exact video in a huge library. They might guess the right answer, but they aren't as sharp as the specialists.

The Solution: VIRTUE

VIRTUE is a new system that takes the "Generalist" (the smart MLLM) and trains it to be a "Specialist" without losing its flexibility. It does this through three main tricks:

1. The "Universal ID Card" (Unified Embeddings)

Imagine every video and every text description gets a unique ID card (an "embedding").

How it works: VIRTUE teaches the AI to look at a video and a text description (like "a cat sleeping") and turn both into the same type of ID card. If the video and the text match, their ID cards look almost identical.
The Magic: Because the AI uses a shared "language" for both pictures and words, it can instantly compare them. It's like having a translator that speaks both "Video" and "Text" fluently, allowing it to find matches instantly without needing to watch every single video.

2. The "Two-Step Search" (Retrieval + Reranking)

Finding the right video in a library of millions is hard. VIRTUE uses a two-step process:

Step 1: The Quick Scan (Embedding Search): The AI quickly scans the whole library using those ID cards to find the top 50 most likely candidates. This is fast and efficient.
Step 2: The Deep Dive (Reranking): Now, the AI takes those top 50 candidates and looks at them very closely, one by one, comparing the specific details of the video against your request. It acts like a senior editor who double-checks the shortlist to make sure the best match is at the very top.
Why it matters: This gives you the speed of a quick scan with the accuracy of a deep review.

3. The "Time Traveler" (Moment Localization)

Sometimes you don't want the whole video; you just want the 5 seconds where the explosion happens.

How VIRTUE does it: Instead of needing special training to find these moments, VIRTUE looks at the video frame-by-frame. It asks, "Does this specific frame match your description?" It then smooths out the answers to find a continuous chunk of time.
The Analogy: It's like watching a movie and pausing exactly when the hero pulls out their sword, without needing to be told specifically how to look for swords. It does this naturally, without needing extra instructions.

The "Change the World" Trick (Composed Queries)

This is where VIRTUE really shines. Imagine you show the AI a video of a sunny beach and say, "Make it snow."

Old Systems: Would get confused. They might just search for "snow" and ignore the beach video, or they might try to edit the video (which they can't do).
VIRTUE: Understands the concept of the change. It looks at the sunny beach video, understands the "snow" instruction, and searches for a video that fits that new mental image. It's like asking a chef, "I want a pizza, but with no cheese," and having them instantly know exactly which pizza to pull from the oven.

Why This Matters

It's Fast: It doesn't need to watch millions of videos to find the right one.
It's Flexible: You can ask simple questions ("Find a cat") or complex ones ("Find a video of a cat, but make it look like it's in space").
It's Efficient: It was trained on a relatively small amount of data (compared to the massive datasets other systems use), yet it beats the "specialist" systems that were trained on huge amounts of data.

In short: VIRTUE is the ultimate video search engine. It combines the speed of a search engine with the intelligence of a human, allowing you to find exactly what you want, whether you describe it with words, pictures, or a mix of both, and whether you want the whole video or just a specific moment.

1. Problem Statement

Current video retrieval systems face a fundamental trade-off between specialization and versatility:

Specialized Systems: Models like InternVideo2 achieve state-of-the-art (SOTA) performance on corpus-level retrieval by training on massive, modality-specific datasets. However, they rely on separate encoders for different modalities, making them incapable of natively processing composed multimodal queries (e.g., "Find a video like this clip, but set in a snowy environment").
Multimodal LLMs (MLLMs): While MLLMs excel at understanding complex, composed queries and temporal reasoning, their direct application to video retrieval has historically underperformed compared to specialized systems. They often lack the efficiency of embedding-based search and struggle with fine-grained temporal localization (moment retrieval) without extensive task-specific fine-tuning.

The Gap: There is no single architecture that simultaneously delivers high-performance corpus-level retrieval, supports zero-shot composed multimodal queries, and enables zero-shot moment localization without sacrificing performance or requiring complex, task-specific architectural changes.

2. Methodology: VIRTUE Framework

VIRTUE is a unified, MLLM-based framework designed to handle corpus-level retrieval, moment localization, and composed querying within a single architecture. It consists of two primary components: VIRTUE-Embed and VIRTUE-Ranker.

A. Core Architecture & Training Strategy

Backbone: The system uses a Qwen2.5-VL (7B) MLLM backbone.
Unified Embedding Mechanism: Instead of generating text, the model is adapted to produce a unified embedding for both queries and videos. This is achieved by using the final hidden state of the End-of-Sequence (EOS) token as the embedding anchor. The causal attention mechanism ensures this token captures a global summary of the multimodal context.
Contrastive Learning (VIRTUE-Embed):
- Objective: The model is trained using InfoNCE loss to align visual and textual embeddings.
- Prompting: A specific prompt ("Summarize this video/text in one word") is appended before the EOS token to force the model to generate a concise, discriminative representation.
- Two-Stage Training:
  1. Image-Text Alignment: Trained on 595K image-caption pairs (CC-595K) to establish broad vision-language priors.
  2. Video-Text Alignment: Fine-tuned on 105K video-text pairs (PE-Video) to adapt to temporal dynamics.
- Efficiency: Uses LoRA (Low-Rank Adaptation) to train only ~48M parameters, keeping the base model frozen.

B. VIRTUE-Ranker (Candidate Refinement)

Purpose: To refine the top- $K$ candidates retrieved by VIRTUE-Embed, addressing the semantic complexity that fixed-dimensional embeddings might miss.
Mechanism: Unlike methods that generate "Yes/No" responses, VIRTUE-Ranker treats re-ranking as a pointwise binary matching problem. It processes the query and candidate video jointly through the MLLM and projects the EOS hidden state through a linear head to predict a confidence score $s \in [0, 1]$ .
Training Objective: A joint loss function combining:
1. Binary Cross-Entropy (BCE): On random and hard negatives.
2. Preference-Based Loss (Bradley-Terry): Optimizes the relative ordering between ground-truth matches and hard negatives, allowing the model to distinguish between partially relevant and irrelevant candidates without forcing absolute scores.

C. Zero-Shot Capabilities

Moment Retrieval: A pipeline that computes frame-level similarity scores between a query and sampled video frames. It applies Gaussian smoothing, peak detection, and Non-Maximum Suppression (NMS) to predict variable-sized temporal windows without any moment-specific training data or architectural changes.
Composed Video Retrieval: The model natively supports queries combining a source video and a text modification (e.g., "change to snowy"). It encodes the interleaved sequence (Video + Text + Instruction) into a single embedding for retrieval, leveraging the MLLM's inherent multimodal reasoning.

3. Key Contributions

Unified Architecture: Introduces VIRTUE, the first framework to unify corpus-level retrieval, moment localization, and composed multimodal querying in a single MLLM-based model.
Performance Parity with Specialized Models: Demonstrates that a retrieval-centric MLLM adaptation can match or exceed specialized systems (like InternVideo2) on standard benchmarks, despite not training on audio data and using significantly less data (~700K pairs vs. hundreds of millions).
Emergent Zero-Shot Capabilities: Achieves SOTA results on zero-shot composed video retrieval (CoVR) and strong performance on zero-shot moment retrieval (Charades-STA, ActivityNet-Captions) without task-specific fine-tuning or external captioning models.
Efficiency: The embedding-only stage requires only one forward pass per query, and the re-ranking stage is computationally efficient (processing only top-50 candidates), avoiding the $O(N \times M)$ cost of joint processing all corpus pairs.

4. Experimental Results

The model was evaluated on multiple benchmarks in a zero-shot setting:

Text-to-Video Retrieval (MSR-VTT, DiDeMo, MSVD):
- VIRTUE-Embed (Embedding-only): Outperforms other MLLM-based embedding models (e.g., VLM2Vec, CaRe) and even surpasses InternVideo2-Embed on DiDeMo (46.6% R@1 vs. 41.8%).
- VIRTUE (with Re-ranker): Achieves SOTA or near-SOTA results. On DiDeMo, it beats the specialized InternVideo2 (58.8% vs. 57.9% R@1). On MSR-VTT, it outperforms LamRA by +7.7 points.
Composed Video Retrieval (CoVR):
- Achieves SOTA performance, outperforming the previous best (Thawakar et al.) by ~8% on R@1 (55.49% vs. 47.52%).
Moment Retrieval:
- Achieves SOTA mIoU on ActivityNet-Captions (33.4) and best R@0.3 on Charades-STA (55.5) in a purely zero-shot setting, outperforming specialized models trained on millions of temporal annotations.

5. Significance and Impact

Paradigm Shift: VIRTUE challenges the notion that specialized architectures are required for high-performance video retrieval. It proves that retrieval-oriented contrastive adaptation of general-purpose MLLMs is a scalable and powerful alternative.
Versatility: It bridges the gap between general multimodal understanding and specific retrieval tasks, enabling complex user interactions (composed queries, temporal localization) that were previously impossible or required separate, disjointed systems.
Efficiency: By leveraging LoRA and a unified embedding space, VIRTUE offers a cost-effective solution that avoids the massive computational overhead of training on hundreds of millions of samples or using complex multi-stage pipelines for every task.
Future Direction: The work suggests that unified retrieval systems can evolve into flexible, general-purpose engines capable of handling increasingly complex and expressive user queries, paving the way for more intelligent video search and recommendation systems.