Technical Summary: Massive Video Embedding Benchmark (MVEB)

Problem Statement

Current video representation benchmarks are fragmented, typically evaluating models on a single capability (e.g., action recognition or retrieval) rather than assessing general-purpose embedding quality. This fragmentation makes it difficult to determine if high scores reflect robust video understanding or overfitting to specific tasks. Furthermore, existing benchmarks often lack standardized protocols for measuring the specific contribution of audio tracks to video understanding, and few support unified evaluation across text, image, audio, and video modalities within a single ecosystem.

Methodology

Benchmark Construction

The authors introduce MVEB (Massive Video Embedding Benchmark), a curated 23-task benchmark derived from a larger pool of 184 tasks (MVEB+). The selection process utilized correlation-based redundancy pruning to maintain task diversity while reducing evaluation costs. MVEB integrates into the existing MTEB (Massive Text Embedding Benchmark) ecosystem, ensuring unified evaluation protocols across text, image, audio, and video.

The benchmark is structured around six task families:

1. Classification: Logistic regression on frozen embeddings (few-shot linear probing).
2. Zero-shot Classification: Matching video embeddings to text prompts without training.
3. Clustering: Evaluating semantic grouping via V-measure.
4. Pair Classification: Predicting binary criteria between video pairs.
5. Retrieval: Ranking corpora by relevance across eight modality directions (e.g., Text→Video, Audio→Video, Audio+Video→Text).
6. Video-Centric Question Answering (VCQA): Selecting relevant answers from text options given video inputs.

A key methodological innovation is the paired evaluation design. For every dataset containing audio, MVEB runs tasks twice: once with video-only input and once with video+audio input. This allows for the direct measurement of audio's contribution based on annotation provenance:

• AV-grounded: Labels produced from both audio and visual content.
• V-grounded: Labels produced from visuals alone (even if audio is present).

Evaluation Protocol

The study evaluates 33 publicly available checkpoints spanning six embedding paradigms:

• Self-supervised Video Encoders (e.g., V-JEPA-2).
• Video-Text Contrastive Encoders (e.g., X-CLIP).
• Audio-Visual Contrastive Encoders (e.g., Perception Encoder).
• Multimodal Binding Models (e.g., eBind).
• MLLM-Based Embedding Models (adapted via contrastive training).
• Generative MLLMs used as embedders (via hidden-state pooling without contrastive adaptation).

Evaluations are conducted zero-shot (no per-task fine-tuning). The protocol respects each model's declared sampling configurations (frame rates, audio duration caps) to avoid out-of-distribution issues.

Key Results

1. No Single Dominant Model

No single model dominates all tasks. Performance varies significantly by paradigm and task type:

• MLLM-based embeddings (e.g., LCO-Embedding-Omni-7B) lead in classification, clustering, pair classification, and QA.
• Multimodal binding models (e.g., eBind) lead in retrieval and zero-shot classification.
• Generative MLLMs used directly as embedders (without contrastive adaptation) collapse on cross-modal tasks, scoring an order of magnitude lower than their adapted counterparts.

2. The Role of Audio

The contribution of the audio track is not universally positive; it depends on the dataset's annotation provenance:

• AV-grounded datasets: Audio provides a positive signal (+0.016 average gain).
• V-grounded datasets: Audio acts as a confound, hurting performance (-0.046 average loss).
  This results in a consistent six-point gap across model families, suggesting audio helps only when the ground truth labels were derived from the audio modality.

3. Modality-Restricted Leaderboards

The paper exposes two sub-leaderboards for models lacking audio encoders:

• MVEB(text, video): Text-video specialists (e.g., Qwen3-VL-Embedding-8B) outperform all audio-capable "omni" models on this subset, reordering the top of the field.
• MVEB(video): Evaluates video-only encoders.

4. Scale vs. Alignment

Smaller specialized models (e.g., eBind at 764M–1.8B parameters) often outperform larger MLLM-based models on retrieval and zero-shot classification. This suggests that training-data alignment (specifically contrastive adaptation) is more critical for cross-modal performance than backbone scale.

Significance and Contributions

The paper claims three primary contributions:

1. A Unified Benchmark: MVEB provides the first comprehensive, 23-task benchmark for video embeddings that integrates with the MTEB ecosystem, covering classification, retrieval, clustering, and QA across text, video, and audio modalities.
2. Actionable Insights for Development: The study establishes that a contrastive embedding stage is a near-prerequisite for cross-modal performance and that training-data alignment matters more than model scale. It also highlights that audio is a distinct signal whose utility is strictly tied to annotation provenance.
3. Community-Driven Maintenance: Unlike static benchmarks, MVEB is designed for long-term maintenance and community contribution, featuring task and model versioning to prevent stagnation.

The authors position MVEB as the "video sibling" to MTEB, MMTEB, MIEB, and MAEB, completing a unified family of embedding benchmarks. The work emphasizes that general-purpose video understanding requires balancing modality-specific strengths (e.g., text-video specialists for text-video tasks) with robust cross-modal alignment, rather than relying on a single "omni" model to solve all tasks equally well.