💬 NLP

UEmbed: Unified Sparse and Dense Multimodal Embeddings

UEmbed introduces a unified decoder-only multimodal embedding model that generates both sparse lexical and dense representations in a single causal forward pass, achieving state-of-the-art performance on multimodal benchmarks while enabling efficient agentic applications.

Original authors: Tingyu Song, Mingxin Li, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Zhijie Nie, Yilun Zhao, Shu Wu

Published 2026-08-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Tingyu Song, Mingxin Li, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Zhijie Nie, Yilun Zhao, Shu Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific book in a massive, chaotic library. For a long time, librarians used a simple trick: they looked for exact words. If you asked for "cat," they only found books with the word "cat" in them. This is fast and easy, but it's a bit dumb; it misses books about "felines" or "kittens" that never used the word "cat." To fix this, scientists invented "smart" search engines that understand meaning. These engines turn your question and the books into long lists of numbers (called "dense vectors"). It's like giving every book a unique fingerprint based on its vibe and story. This is great for understanding, but it's heavy, slow, and sometimes hard to explain why a book was chosen.

Now, imagine a new kind of librarian who can do both jobs at once. They can give you the "vibe fingerprint" and a list of the most important keywords, all in a single, lightning-fast glance. This is the world of information retrieval, the science of finding needles in digital haystacks. The big question researchers have been asking is: Can we build a single, super-smart brain that handles both the "exact word" search and the "meaning-based" search without needing two different, clunky systems? And can this brain understand not just text, but also pictures, videos, and documents all at once? This is where the story of a new invention called UEmbed begins.

The One-Brain Solution: UEmbed

Meet UEmbed (Unified Embedding), a new kind of computer brain designed by researchers from Alibaba, the Chinese Academy of Sciences, and Yale. Think of UEmbed as a multitasking superhero who refuses to choose between being a "word wizard" and a "meaning master." In the past, if you wanted a search engine that understood deep meanings, you had to use a heavy, slow system. If you wanted one that was fast and used keywords, you had to use a simpler, dumber system. Usually, you couldn't have both in the same package, especially when dealing with images and videos.

UEmbed changes the game by using a "decoder-only" architecture. To use a simple analogy, imagine a standard search engine is like a person reading a book from start to finish, looking back and forth to understand the whole story (this is called "bidirectional"). UEmbed, however, is like a storyteller who listens to the whole story and then, at the very end, instantly summarizes it in two ways: first, by giving you a "vibe score" (the dense part), and second, by shouting out the top 10 to 20 most important words that popped into their head (the sparse part). It does all this in one single pass, like flipping a switch.

How It Works: The Magic of Special Tokens

So, how does this brain manage to shout out keywords while also understanding the whole story? The secret sauce is a clever trick involving "special tokens."

Imagine you are writing a story, and at the very end, you attach a few invisible, magical sticky notes. In UEmbed's case, the researchers attach 16 of these special notes (called tokens) to the end of the input. Before the computer starts reading, it divides the entire dictionary of words (the vocabulary) into 16 different groups, like sorting a giant box of LEGO bricks into 16 different colored bins.

As the computer reads your image, video, or text, it processes the story. When it reaches those 16 magical sticky notes at the end, each note is assigned a specific job: "You are in charge of the red bin of words," "You handle the blue bin," and so on. Each note looks at the whole story it just heard and decides, "Based on what I heard, these are the most important words from my color bin."

By the time the computer is done, it has 16 small lists of important words. It stitches them all together to create a massive, super-detailed list of keywords (the sparse vector). At the same time, it takes the "vibe" of the whole story to create the dense fingerprint. This clever partitioning solves a major problem: usually, a computer brain can only use one "summary token" to describe a whole story, which isn't enough to hold all the keywords. By using 16 different tokens, UEmbed spreads the work out, allowing it to be both keyword-rich and meaning-rich at the same time.

What the Numbers Say

The researchers tested UEmbed on some of the toughest challenges in the world of search. They didn't just look at text; they threw images, videos, and complex documents at it.

The Big Score: On a massive benchmark called MMEB-v2, which tests how well models understand different types of media, the biggest version of UEmbed (the 9B model, which has 9 billion parameters) scored 71.8 for its "vibe" search and 71.0 for its "keyword" search.
The Comparison: This is a huge deal. Usually, keyword-based search lags far behind "vibe" search. Here, UEmbed's keyword mode is almost as good as its vibe mode, and it beats almost every other model that was trained on public data. For example, it outperformed a 7-billion-parameter model called RzenEmbed-V2-7B (which scored 71.1) and a 2-billion-parameter model called Embed-RL-4B (which scored 68.1).
The Efficiency: Because UEmbed is built like a "decoder-only" model (the same kind used by popular chatbots), it plays nicely with high-speed servers. It can generate these dual results incredibly fast, making it practical for real-world use.

Why This Matters: The "Agent" Advantage

The paper suggests that this dual-power approach isn't just a neat trick; it's a practical necessity for the future of AI "agents"—smart programs that browse the web and do tasks for you.

When an AI agent is trying to solve a problem, it often asks short, keyword-heavy questions like "best pizza near me" or "how to fix a leak." Dense "vibe" search sometimes misses these because it looks for deep meaning, while keyword search is fast but dumb. UEmbed offers the best of both worlds. In tests on a benchmark called BrowseComp-Plus, the researchers found that using UEmbed's keyword mode helped the AI agent find the right information with fewer search attempts, saving time and computing power.

The Limits and the Future

The authors are careful to point out that UEmbed isn't perfect yet. They noticed that the model sometimes gets a bit confused with languages other than English and Chinese, likely because it was mostly trained on those two. They also saw that sometimes the "magic sticky notes" might shout out weird, non-standard words (like "_alt" or "_perm"), which are artifacts of the computer's internal dictionary.

However, the core idea is solid: UEmbed proves that you don't need to choose between speed and smarts, or between words and meaning. By unifying these two worlds into a single, decoder-only model, it opens the door for search engines that are faster, smarter, and capable of understanding the entire digital world—from a single sentence to a full-length video—in one go. It's a new paradigm where the search engine doesn't just look for words or just feel the vibe; it does both, instantly.

Technical Summary: UEmbed

Problem Statement

Modern information retrieval systems increasingly rely on Learned Sparse Retrieval (LSR) to move beyond exact lexical matching toward richer semantic understanding. However, existing LSR methods face three critical limitations:

Architectural Constraints: Current LSR models are predominantly built on bidirectional encoder architectures (e.g., BERT). They cannot natively leverage the causal attention mechanisms of decoder-only Large Language Models (LLMs) and Multimodal LLMs (MLLMs) without converting the backbone to bidirectional behavior or adopting complex training curricula, which increases costs and breaks compatibility with optimized serving frameworks like vLLM.
Limited Modality: Most LSR approaches are text-only. Multimodal extensions typically rely on auxiliary cross-modal modules rather than deriving sparse representations directly from a unified multimodal backbone, making them difficult to extend to new modalities.
Unexplored Utility: The practical advantages of sparse retrieval (e.g., efficiency, interpretability, agentic search utility) remain under-examined beyond standard accuracy benchmarks.

A specific technical challenge in adapting sparse retrieval to causal models is the information bottleneck: relying on a single token (e.g., EOS) to project into a massive vocabulary space severely limits representational capacity.

Methodology: UEmbed

The authors propose UEmbed (Unified Embedding), a decoder-only multimodal embedding model that natively produces both sparse lexical and dense representations in a single causal forward pass.

Core Architecture

UEmbed modifies a standard decoder-only MLLM by appending $N$ learnable special tokens ( $\langle s_1 \rangle, \dots, \langle s_N \rangle$ ) to the input sequence.

Vocabulary Partitioning: The vocabulary $V$ is partitioned into $N$ disjoint subsets ( $V_1, \dots, V_N$ ) using $k$ -means clustering. This encourages each special token to act as a "soft topic specialist" for a distinct semantic subspace.
Partitioned Sparse Heads: Each special token $\langle s_k \rangle$ summarizes the preceding input via causal attention and predicts sparse weights over its assigned vocabulary subset $V_k$ using a linear projection:
$w^{(k)}_t = \log(1 + \text{ReLU}(W^{(k)\top}_t h_{s_k} + b^{(k)}_t))$
The final sparse vector is the concatenation of all subset vectors.
Dense Representation: The dense embedding is derived from the hidden state of the EOS token immediately preceding the special tokens. If only dense retrieval is needed, the special tokens can be omitted entirely, incurring no extra computation.

Training Strategy

Data: Trained on a mixture of 3.94M public samples covering text, images, videos, and visually-rich documents (sources include Echo-embedding, MLDR, and MMEB).
Hard Negative Mining: For multimodal data lacking negatives, a teacher model (Qwen3-VL-Embedding-8B) is used to mine hard negatives.
Unified Objective: The model is trained with a combined loss:
$L = L^{\text{dense}}_{\text{InfoNCE}} + \lambda L^{\text{sparse}}_{\text{InfoNCE}} + \alpha_q L^q_{\text{FLOPS}} + \alpha_d L^d_{\text{FLOPS}}$
The FLOPS regularizers encourage sparsity by penalizing the squared mean term weight.

Key Contributions

Unified Decoder-Only Retrieval: UEmbed is the first model to produce both sparse and dense retrieval representations from a single decoder-only backbone, eliminating the need for bidirectional encoders or auxiliary cross-modal modules.
State-of-the-Art Sparse Multimodal Retrieval: To the authors' knowledge, UEmbed is the first sparse model evaluated on MMEB-v2, establishing a new state of the art in sparse multimodal retrieval.
Practical Utility Demonstration: The paper demonstrates that UEmbed's sparse mode improves efficiency (native inverted-index compatibility), effectiveness (via hybrid scoring), and utility in agentic search (reducing tool-call costs).

Experimental Results

UEmbed was evaluated at 2B, 4B, and 9B parameter scales.

Multimodal Benchmarks (MMEB-v2):
- UEmbed-9B achieves 71.8 (dense) and 71.0 (sparse), leading all models trained on public data.
- The gap between dense and sparse performance is minimal (within 1.0 point across all scales), proving sparse embeddings are a viable alternative in the multimodal domain.
- UEmbed-4B (sparse, 69.7) outperforms the dense Ops-MM-Embed-7B (67.1).
Text Benchmarks (BEIR):
- UEmbed-9B achieves an average nDCG@10 of 56.3 in dense mode and 55.2 in sparse mode.
- In sparse mode, UEmbed-9B matches the performance of the specialist model Echo-Mistral-SPLADE (55.2) while simultaneously supporting dense retrieval and multimodal inputs.
Agentic Search (BrowseComp-Plus):
- The sparse mode consistently requires fewer search rounds (tool calls) than the dense counterpart while maintaining comparable or higher recall, validating its cost-effectiveness for iterative reasoning.

Significance and Claims

The paper positions UEmbed as a paradigm shift that unifies dense and sparse embeddings within a single model, extending sparse retrieval to unify text and multimodal inputs.

Architectural Efficiency: By operating natively under causal attention, UEmbed retains compatibility with high-throughput serving stacks (e.g., vLLM) and standard inverted-index search, offering deployment advantages over bidirectional LSR models.
Effectiveness: The model demonstrates that sparse retrieval can achieve performance parity with dense retrieval in multimodal settings, challenging the assumption that dense vectors are strictly superior for complex modalities.
Interpretability: The sparse mode provides inherent interpretability by surfacing activated vocabulary terms, which aids in bias auditing and understanding model decisions.

The authors acknowledge limitations, including language bias (predominantly English/Chinese), occasional generation of anomalous tokens due to large vocabulary sizes, and slightly larger performance gaps in video retrieval compared to static documents. However, they conclude that UEmbed successfully transforms sparse retrieval from a legacy standalone module into a native byproduct of MLLMs.