🤖 AI

MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarios

This paper introduces MMLongEmbed, the first comprehensive benchmark designed to evaluate Multimodal Embedding Models in long-context scenarios, revealing that current state-of-the-art models struggle with deep semantic understanding and exhibit significant performance degradation as context length increases.

Original authors: Haitian Wang, Ruoxi Sun, Quantong Qiu, Juntao Li, Junhui Li, Hua Chen, Jinxiong Chang, Min Zhang

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haitian Wang, Ruoxi Sun, Quantong Qiu, Juntao Li, Junhui Li, Hua Chen, Jinxiong Chang, Min Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a library that can hold an entire encyclopedia, a thousand movies, and a million web pages all at once. Now, imagine you ask a librarian (an AI) to find one specific sentence about a blue cat hidden somewhere in that massive pile.

This paper, MMLongEmbed, is essentially a report card for these "super-librarians" (called Multimodal Embedding Models) when they are forced to work with these giant, long piles of information.

Here is the breakdown of what the researchers found, using simple analogies:

1. The Problem: Big Library, Small Brain?

Recently, AI models have gotten "bigger" in terms of how much text and images they can read at once (their "context window"). It's like giving a librarian a bigger desk. But the researchers asked: Just because they have a bigger desk, does that mean they can actually find the needle in the haystack?

They found that the answer is often no. Even though these models can see the whole library, they often struggle to understand the deep connections inside it. They tend to rely on "surface-level" clues (like matching a word here or a color there) rather than truly understanding the story.

2. The Test: MMLongEmbed

To test this, the authors built a new exam called MMLongEmbed. Think of it as a "Survival of the Fittest" test for AI librarians. They created four specific challenges:

The "Needle in a Haystack" (Visual Grounding): They hid a specific fact (a "needle") inside a massive document. The test checks if the AI can find it whether it's at the very beginning, the very end, or buried in the middle.
- The Result: The AI is great at finding things at the start or end of a document but often gets "lost" in the middle, like a reader who only remembers the first and last page of a book.
The "Detective Puzzle" (Multi-Source Reasoning): They gave the AI a question that required connecting clues from different documents (e.g., "Who played in this movie and also wrote this song?").
- The Result: The AI gets confused when the clues are buried deep in long texts.
The "Book Report" (Logical Summarization): They asked the AI to summarize a 30,000-word document into a short paragraph.
- The Result: The AI is actually pretty good at this! Because summaries rely on general themes, the AI can "guess" the main idea even if it misses some details.
The "Time Traveler" (Temporal Summarization): They showed the AI a long video and asked, "What happened first, and what happened next?"
- The Result: The AI struggles badly here. If you shuffle the order of events in the video, the AI often fails to notice the timeline is broken.

3. The Tricky Part: The "Distractor" Trap

The researchers made the test harder by adding distractors. Imagine you are looking for a red car in a parking lot.

Easy Mode: The parking lot has one red car and 100 blue cars. (The AI finds it easily).
Hard Mode: The parking lot has 100 red cars that look almost exactly the same, but one has a slightly different bumper. (The AI gets confused and picks the wrong one).

The paper found that when the AI faces these "Hard Mode" distractors, its performance crashes. This proves the AI is often just matching keywords (like "red car") rather than understanding the specific details (like "the car with the dented bumper").

4. The Surprising Findings

Bigger isn't always better: Making the AI model larger (more parameters) or giving it a bigger memory (more dimensions) didn't consistently fix the problem. A slightly smaller, smarter model sometimes outperformed a giant one.
The "Middle" is the danger zone: The AI has a strong bias toward the beginning and end of a document. If the answer is in the middle of a 30,000-token text, the AI often forgets it exists.
Video is hard: The AI is much worse at understanding the order of events in a video compared to just recognizing what is in the video.

5. The Conclusion

The paper concludes that while we have built AI that can read massive amounts of data, we haven't yet taught it how to think deeply about that data when it's buried in a long context.

Currently, these models are like students who can memorize the first and last sentence of a long essay perfectly but lose the plot in the middle. To make them truly useful for real-world tasks (like finding specific evidence in a legal case or a medical record), we need to teach them to pay attention to the deep structure of the information, not just the surface words.

In short: We gave the AI a bigger library, but it still needs to learn how to read the books properly, not just skim the covers.

Technical Summary: MMLongEmbed

Problem Statement

Recent advancements in Multimodal Embedding Models (MEMs) have significantly expanded theoretical context windows to 32K tokens and beyond, enabling the unified representation of interleaved text, document images, and videos. However, the paper argues that larger context windows do not necessarily translate into effective comprehension or representation of long-context multimodal inputs. This gap remains a critical bottleneck for real-world deployment in applications such as multimodal retrieval-augmented generation (RAG) and autonomous agent workflows.

Current evaluation frameworks suffer from three primary deficiencies:

Limited Context Diversity: Existing benchmarks are largely restricted to text-only data, failing to cover the multimodal information encountered in real-world scenarios.
Lack of Length Stratification: Input length is often not treated as a controlled variable, preventing systematic analysis of performance degradation as context scales.
Insufficient Task Difficulty: Many benchmarks rely on simple candidate pool expansions where tasks can be solved via shallow feature matching, lacking the depth required to test true semantic understanding.

Methodology: MMLongEmbed Benchmark

To address these gaps, the authors introduce MMLongEmbed, the first comprehensive benchmark designed to systematically evaluate MEMs in long-context scenarios.

Benchmark Construction

The benchmark integrates eight diverse datasets comprising 8,460 queries and 20,880 candidates, supporting context lengths up to 32K tokens. It supports three input formats: image-only, interleaved image–text, and text-only. The evaluation is structured around four distinct tasks:

Visual Grounding (Synthetic): A "Needle-in-a-Haystack" framework using data from NeedleBench and MMNIAH. Target instances ("needles") are embedded into multimodal haystacks sampled from OBELICS. The setup varies insertion depths (0% to 100%) and context lengths (256 to 32K tokens) across eight intervals. Crucially, distractors are designed to be highly similar to the target to introduce fine-grained retrieval ambiguity.
Multi-Source Reasoning (Real-World): Built upon HotpotQA and MMDocRAG, this task assesses the ability to capture structured evidence across heterogeneous contexts. The benchmark expands original contexts to 16K and 32K tokens by injecting semantically irrelevant but naturally flowing text. "Hard" distractors are generated by removing critical evidence spans, breaking reasoning chains while preserving surface-level topical similarity.
Logical Summarization (Real-World): Using GovReport and MMLongBench, this task evaluates global semantic compression. Hard distractors are created by replacing large contiguous spans (e.g., first 25%, last 25%, or middle 50%) with query-irrelevant content sampled from topically similar documents. This forces models to rely on logical flow rather than localized keyword matching.
Temporal Summarization (Real-World): Sourced from ActivityNet Captions and YoukuDenseCaption, this task evaluates the understanding of long-range temporal structures in video. Hard distractors involve macro-structural transformations such as chronological reversal and block-level permutation, disrupting global narrative order while preserving event-level semantics.

Tokenization Standardization

To ensure consistent length intervals across models with varying visual tokenization schemes, the authors adopt a standardized counting mechanism:

Text Tokens: Computed using the Qwen3 tokenizer.
Visual Tokens: Images are processed with a $16 \times 16$ patch embedding followed by a $2 \times 2$ spatial merging, resulting in an effective patch size of $32 \times 32$ pixels per token. The token count is calculated as $\lceil H/32 \rceil \times \lceil W/32 \rceil$ .

Evaluation Protocol

The benchmark evaluates 11 leading MEMs (10 open-source, 1 proprietary), ranging from 2B to 8B parameters, with context windows of at least 32K. The evaluation strictly controls candidate density (Avg. Candidates per Query) across length pools and employs a "middle truncation" strategy to prevent out-of-memory errors for inputs exceeding specific limits.

Key Results

Extensive experiments reveal significant limitations in current MEMs regarding long-context multimodal understanding:

Decoupling of Scale and Capability: Increasing model size or embedding dimensionality does not consistently improve performance. In some cases, smaller models outperform larger counterparts within the same family (e.g., Qwen3-2B vs. Qwen3-8B in specific hard subsets). The modeling paradigm and training strategy appear more decisive than scale.
Positional Bias and Information Loss: Performance on fine-grained retrieval (Visual Grounding) exhibits severe degradation for targets located in intermediate positions, with a strong bias toward sequence boundaries. Most models maintain stable retrieval only up to ~2K tokens, after which performance deteriorates rapidly regardless of model size.
Reliance on Superficial Matching: Models perform well on "Easy" tasks but suffer sharp performance drops when superficial matching cues are suppressed by high-similarity distractors ("Hard" tasks). This indicates a heavy reliance on global semantic alignment rather than deep fine-grained discrimination.
Task-Specific Degradation:
- Summarization Tasks: Performance remains relatively stable across lengths, likely due to high redundancy and distributed evidence.
- Reasoning and Temporal Tasks: Performance shows a consistent decline as context length increases. Temporal summarization is particularly fragile, collapsing to near-random levels under hard distractors.
Modality Robustness: Models exhibit varying degrees of robustness to heterogeneous background interference. While some models show modality skew (uneven performance across text/image distractors), a small subset demonstrates modality-robust behavior, maintaining stable performance across all interference types.

Significance and Claims

The paper claims that MMLongEmbed bridges a critical gap in the evaluation of multimodal embedding models. By systematically varying context length, modality, and task difficulty, the benchmark exposes that current architectures rely heavily on surface-level feature matching and struggle to capture deep semantic and structural dependencies in long contexts.

The authors conclude that simply scaling context length, model size, or embedding dimensionality is insufficient to address these limitations. Instead, the findings underscore the need for fundamentally improved long-context semantic modeling that can effectively handle fine-grained information retention and deep structural reasoning across multimodal inputs. The benchmark and code are publicly available to facilitate reproducibility and further research.