💻 computer science

Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini

The paper introduces Gemini Embedding 2, a native multimodal embedding model that unifies video, audio, image, and text into a single representation space, achieving state-of-the-art performance across diverse unimodal, cross-modal, and multimodal retrieval tasks while demonstrating robust zero-shot capabilities in specialized domains.

Original authors: Madhuri Shanbhogue, Zhe Li, Shanfeng Zhang, Gustavo Hernández Ábrego, Shih-Cheng Huang, Aashi Jain, Daniel Salz, Sonam Goenka, Chaitra Hegde, Ji Ma, Feiyang Chen, Jiaxing Wu, Tanmaya Dabral, Babak Sam

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Madhuri Shanbhogue, Zhe Li, Shanfeng Zhang, Gustavo Hernández Ábrego, Shih-Cheng Huang, Aashi Jain, Daniel Salz, Sonam Goenka, Chaitra Hegde, Ji Ma, Feiyang Chen, Jiaxing Wu, Tanmaya Dabral, Babak Samari, Kevin Poulet, Daniel Cer, Kaifeng Chen, Paul Suganathan, Hui Hui, Jovan Andonov, Philippe Schlattner, Jay Han, Iftekhar Naim, Wing Lowe, Vladimir Pchelin, Albert Yang, Yi-Ting Chen, Zhongli Ding, Grace Zhang, Georg Heigold, Yichang Chen, Antoine Reveillon, Brendan Mccloskey, Wenlei Zhou, Dahun Kim, Rui Meng, Emma Wang, Jack Zheng, Halley Fede, Zhen Yang, Keegan Mosley, Brian Potetz, Sahil Dua, Henrique Schechter Vera, Shen Gao, Hesen Zhang, Andreas Hess, Hengxuan Ying, Alberto Montes, Karan Gill, Min Choi, Sebastian Russo, Anja Hauth, Jinhyuk Lee, Michael Boratko, Megan Barnes, Vikram Rao, Claudiu Musat, Cyril Allauzen, Ehsan Variani, Shankar Kumar, Tom Bagby, Junyi Jiao, Yang Gu, Tengxin Li, Ayush Agrawal, Roberto Santana, Dev Nath, Stephen Karukas, Shuoxuan Han, Lucia Loher, Alice Twu, Nidhi Vyas, Siddharth Bhai, Frank Palma Gomez, Wangyuan Zhang, Chaoren Liu, Jizheng Yang, Steve Qiu, Shijie Zhang, Sujay Kulkarni, Sascha Rothe, Sean Nakamoto, Raphael Hoffmann, Zach Gleicher, Yunhsuan Sung, Qin Yin, Tom Duerig, Mojtaba Seyedhosseini

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library containing books, movies, music recordings, and photographs. Right now, if you want to find a specific song that sounds like a movie scene, or a recipe that matches a photo of a dish, you usually have to translate everything into text first. You'd have to write a description of the photo, transcribe the audio into words, and then search. It's like trying to find a specific color by only looking at a list of paint names; you lose the actual "feel" of the color.

Gemini Embedding 2 is Google's new tool that changes the game. Instead of forcing everything into text, it creates a single, universal "language" where pictures, sounds, videos, and words all speak the same dialect.

Here is a breakdown of how it works and why it matters, using simple analogies:

1. The Universal Translator (The Core Idea)

Think of the old way of doing things as having different dictionaries for different languages. If you wanted to compare a French book to a German movie, you had to translate both into English first, which often lost the nuance.

Gemini Embedding 2 is like a universal translator that speaks "Meaning."

It takes a video, a song, a photo, or a paragraph of text and converts them all into a single type of "ID card" (a mathematical vector).
Because they are all in the same "language," the model can instantly see that a video of a dog barking and a text saying "a loud dog" are related, even though one is sound and the other is words. It doesn't need to convert the sound into words first; it understands the sound directly.

2. The "All-in-One" Chef

Previous models were like chefs who could only cook one type of food well. One chef was great at text, another at images, and another at video. If you wanted a meal with all three, you had to hire three different chefs and hope they agreed on the flavor.

Gemini Embedding 2 is a master chef who can handle any ingredient.

It was trained on a massive mix of tasks: reading code, understanding movies, listening to audio, and analyzing documents.
Because it learned from this huge variety, it doesn't get confused when you mix ingredients. You can ask it to find a video clip using a mix of a photo and a sentence, and it understands the combination perfectly.

3. The "Zero-Shot" Superpower

Usually, if you want a computer to understand a very specific topic (like astronomy or cooking), you have to teach it specifically for that topic. It's like hiring a tutor to teach a student only about the solar system.

Gemini Embedding 2 is like a student who is already an expert in everything.

The paper shows that this model works incredibly well on specialized topics like microscopy (biology), astronomy, fine art, and cooking without needing any extra training.
It's "out-of-the-box" ready. If you show it a picture of a star chart or a complex recipe, it immediately understands the context better than specialized models that were only trained on one of those things.

4. The Audio Breakthrough (No More Transcription)

One of the biggest problems with searching audio is that computers usually have to "read" the audio first (transcribing speech to text) before they can search it. This is like trying to find a song by reading the lyrics, but if the singer mumbles or the accent is thick, the computer gets the lyrics wrong and you never find the song.

Gemini Embedding 2 skips the middleman.

It listens to the raw sound directly. It understands the tone, the pitch, and the emotion of the voice, not just the words.
The paper found that searching with raw audio is much more accurate than searching with transcribed text. It's like recognizing a friend's voice in a crowd without needing to hear what they are saying.

5. How It Was Built (The Training Recipe)

To build this "universal translator," the team didn't just teach it one thing at a time. They used a three-step cooking process:

Pre-Fine-Tuning: They gave the model a huge, messy pile of data (text, code, images) to get it used to the idea of "encoding" information.
Fine-Tuning: They gave it very specific, high-quality examples of how to match things together (like matching a question to an answer, or a video to a description).
Model Souping: This is a clever trick where they took several different versions of the trained model and "blended" them together, like mixing different batches of soup to get the perfect flavor. This made the final model more stable and better at handling different types of tasks at once.

The Bottom Line

Gemini Embedding 2 is a powerful new engine that lets computers understand the world the way humans do: by seeing, hearing, and reading everything as a connected whole. Whether you are searching for a specific moment in a video using a text description, finding a song based on a hummed tune, or looking up a document that contains charts and text, this model does it all in one unified space, often better than specialized tools designed for just one of those jobs.

Technical Summary: Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini

1. Problem Statement

Existing multimodal embedding models, such as CLIP, ALIGN, and SigLIP 2, typically rely on "late-fusion" approaches. These models use paired cross-modal data to train modality-specific encoders (e.g., separate image and text encoders) that map into a unified vector space. While effective for unimodal and simple cross-modal tasks, this architecture has a critical limitation: it struggles to handle mixed-modality inputs (e.g., interleaved sequences of text, images, and video) and fails to fully utilize the rich semantic interactions between modalities. Furthermore, traditional pipelines for audio often require intermediate transcription (ASR) before embedding, introducing error propagation and losing acoustic nuances like prosody and intonation.

The paper posits that with the advent of Multimodal Large Language Models (MLLMs), it is now possible to create native multimodal embeddings that deeply fuse cross-modal interactions, enabling a single model to process arbitrary combinations of text, image, audio, and video in a coherent, unified representation space.

2. Methodology

Model Architecture

Gemini Embedding 2 is built upon the Gemini foundation model, leveraging its native multimodal capabilities.

Input Processing: Unlike previous models requiring separate encoders, Gemini Embedding 2 utilizes Gemini's internal capabilities to convert raw inputs (images, video, audio, text) directly into token sequences. This allows the model to natively accept heterogeneous inputs without intermediate modalities (like ASR for audio).
Encoder: The model employs a transformer with bidirectional attention (initialized from Gemini) to process the token sequence $T$ of length $L$ . This overcomes the unidirectional attention limits of causal LLMs, enabling context-aware reasoning across the entire input sequence.
Pooling and Projection: The token embeddings are aggregated using mean pooling along the sequence axis. A linear projection layer then scales the result to the target dimension. The model outputs 3,072-dimensional vectors, with support for Multi-Resolution Loss (MRL) to optimize for 768 and 1,536 dimensions as well.

Training Objective

The model is trained using a multi-task, multi-stage approach to learn a unified embedding space:

Loss Function: The primary objective is Noise-Contrastive Estimation (NCE) with in-batch negatives. The loss function maximizes the similarity between a query $q_i$ and a positive target $p^+_i$ while minimizing similarity to hard negatives $p^-_i$ and other batch items.
Task Strings: For text-only tasks, prescribed task strings (e.g., "fact checking") are included but randomly dropped during training to ensure robustness across modalities where such strings are absent.
MRL: To support flexible embedding dimensions, the loss is adapted using Multi-Resolution Loss (MRL) across overlapping sub-dimensions.

Training Recipe

The training process consists of three distinct stages:

Pre-Fine-Tuning (PFT): Adapts the model from auto-regressive generation to encoding. This stage uses large batch sizes and a multi-task setup with image, text, and code tasks to stabilize gradients against noisy inputs.
Fine-Tuning (FT): A comprehensive stage involving text, code, document, image, audio, and video tasks. This stage utilizes query-target-hard negative triplets and samples from single tasks to build batches, carefully balancing sampling rates to optimize performance across all modalities.
Model Soup: To enhance generalization, the final model is created by averaging the parameters of individual fine-tuning checkpoints (including different training runs and weighted averages).

3. Key Contributions

Native Multimodal Embedding: Introduces the first model capable of embedding arbitrary combinations of interleaved text, image, audio, and video into a single vector space, moving beyond the limitations of late-fusion architectures.
Superior Audio Understanding: Demonstrates that native audio processing (directly encoding raw audio) significantly outperforms traditional pipelines that rely on ASR transcription, preserving acoustic cues and reducing error propagation.
Unified Training Framework: Successfully unifies text, code, image, video, and audio training into a single model, achieving state-of-the-art performance across diverse benchmarks without sacrificing performance in any single modality.
Robust Zero-Shot Generalization: Shows strong performance in specialized domains (astronomy, bioscience, culinary arts, fine arts) without domain-specific fine-tuning, establishing the model as a reliable "out-of-the-box" representation.

4. Experimental Results

Multimodal Retrieval Benchmarks

Gemini Embedding 2 achieves state-of-the-art (SOTA) performance across a wide range of retrieval tasks:

Image-Text Retrieval: Achieves 62.9 R@1 on MSCOCO and 89.1 R@1 on Flickr30k (Image-to-Text), surpassing models like Voyage-3.5-multimodal and Amazon Nova MME.
Text-Video Retrieval: Scores 68.8 NDCG@10 on Vatex and 68.0 NDCG@10 on MSR-VTT.
Document Retrieval: Achieves 64.9 NDCG@10 on ViDoRe V2, demonstrating strong capability in understanding page-level visual structures and layouts.
Overall: The model leads decisively in unimodal image retrieval, text-to-image, image-to-text, and text-to-video tasks.

Text and Code Embedding (MMTEB)

Multilingual Performance: Achieves 69.9 on the MTEB Multilingual benchmark, outperforming previous text-only Gemini Embedding models and other multimodal competitors.
Code Retrieval: Sets a new SOTA on MTEB Code v1 with a score of 84.0, significantly outperforming domain-specific models like voyage-code-3.

Specialized Domains (Zero-Shot)

In zero-shot evaluations across specialized fields, Gemini Embedding 2 consistently outperforms baselines:

Microscopy (MicroVQA): 79.3 R@5 (vs. 44.4 for CLIP-Large).
Astronomy (AstroLLaVA): 64.4 R@5 (vs. 31.6 for CLIP-Large).
Culinary Arts (Recipe1M): 90.2 R@5 for ingredients and 92.1 R@5 for instructions.
Fine Art (ArtCap): 67.7 R@5.

Audio Retrieval (MSEB)

On the Massive Sound Embedding Benchmark (MSEB), the model with native audio input achieves an average mrr@10 of 73.99, compared to 70.40 for the ASR-based baseline.

Intra-lingual: +2.0 point improvement.
Cross-lingual: +5.01 point improvement, validating that the model aligns semantic features across languages without relying on phonetic bounds of an intermediate transcriber.

5. Significance and Claims

The paper claims that Gemini Embedding 2 represents a transformative step in general-purpose representation learning. Its significance lies in:

Eliminating Modality Barriers: By natively processing interleaved inputs, it enables complex retrieval paradigms (e.g., finding specific temporal events in a video using combined visual and textual prompts) that were previously difficult or impossible.
Efficiency and Robustness: It removes the need for costly, error-prone intermediate conversions (like ASR) and manual prompt engineering, offering a robust, unified latent space that performs well in zero-shot settings.
Enterprise Readiness: The model is positioned as a highly reliable candidate for downstream enterprise use cases, including Retrieval-Augmented Generation (RAG), video recommendations, and search, capable of handling the heterogeneous data rich in modern applications.
Cross-Domain Consistency: Unlike specialized models that fluctuate in performance across domains, Gemini Embedding 2 maintains a consistent, general-purpose alignment, making it suitable for diverse fields ranging from bioscience to the culinary arts.

The authors conclude that this unified approach facilitates the construction of next-generation agentic systems, providing the essential infrastructure for sophisticated cross-data retrieval without the constraints of modality-specific architectures.