🤖 AI

Can Frontier LLMs Match Natively Multimodal Embeddings? A Comparison on Hard-Negative Text-to-Image Retrieval

This study demonstrates that frontier LLMs like GPT-4.1 and Claude Sonnet 4.6 achieve retrieval performance on par with Google's native multimodal Gemini Embedding 2 on Flickr30k, though the latter remains superior for low-latency applications due to precomputable embeddings.

Original authors: Archan Dutta, Vyanktesh Kanungo

Published 2026-08-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Archan Dutta, Vyanktesh Kanungo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific needle in a massive, chaotic haystack. But here's the twist: the needle isn't just a piece of metal; it's a picture, and you are describing it using words. This is the world of multimodal retrieval, a branch of computer science where machines learn to connect what they see (images) with what they read (text). For years, the standard way to do this was to build two separate "brains": one that reads text and turns it into a secret code, and another that looks at pictures and turns them into a different secret code. The computer then tries to match these two codes, like trying to fit a square peg into a round hole by squinting really hard.

Recently, two new super-tools have entered the arena. First, there are Frontier Large Language Models (LLMs). Think of these as incredibly smart, all-knowing librarians who can look at a pile of 25 photos and a single sentence, then instantly read every photo, compare them to each other, and pick the best match. Second, there are Natively Multimodal Embeddings. These are like a universal translator that speaks both "Image" and "Text" fluently from the start, turning everything into a single, unified secret code without needing two separate brains to talk to each other. The big question everyone is asking is: Do we need the super-smart librarian to read every single photo and compare them, or is the universal translator fast and accurate enough to do the job alone? This matters because if the librarian is too slow, you can't use them for real-time apps like searching your photo gallery while walking down the street.

This paper sets up a high-stakes race between these two approaches using a dataset called Flickr30k, which contains 31,000 images with human-written descriptions. To make the race truly fair and difficult, the researchers didn't just throw random photos at the models. They created "hard negatives"—photos that look very similar to the correct answer but aren't quite right, designed to trick even the smartest systems. They tested four heavyweights: Gemini Embedding 2 and Amazon Nova 2 (the universal translators) against GPT-4.1 and Claude Sonnet 4.6 (the super-smart librarians).

The results were a shock to the speedsters but a relief to the accuracy buffs. When it came to getting the right answer, the universal translators and the super-smart librarians were neck-and-neck. The study found that Gemini Embedding 2, GPT-4.1, and Claude Sonnet 4.6 performed statistically the same. They were so close that you couldn't tell them apart based on accuracy alone; they all managed to pick the correct image about 80% of the time in these tricky scenarios. The paper explicitly rules out the idea that the librarians are vastly superior in accuracy, showing that the new embedding models are just as sharp at finding the right needle in the haystack. However, one model, Amazon Nova 2, did lag behind, getting the right answer about 13 percentage points less often, likely because it was set up for general tasks rather than this specific type of image search.

But the real story isn't just about who wins the accuracy race; it's about who finishes the marathon. Here is where the two approaches diverge wildly. The super-smart librarians (LLMs) have to look at every single photo in the group for every single question. The paper measured this and found that ranking 1,000 queries took the librarians over 6,100 seconds (for GPT-4.1) and 9,415 seconds (for Claude Sonnet 4.6). That's hours of waiting. In contrast, the universal translators (embedding models) work differently. They can pre-calculate the secret codes for all the photos once, like stamping every book in a library with a barcode before you even walk in. Once that "pre-computation" is done, finding the right photo for 1,000 queries took the embedding models less than 2 seconds.

The authors conclude that while the super-smart librarians are brilliant at reasoning and comparing images side-by-side, the universal translators are the clear winners for anything that needs to happen instantly. If you are building an app where a user needs to search a large collection of images in the blink of an eye, the embedding models are the better choice. The librarians are still useful if your photo collection changes every second or is very small, making the pre-calculation step useless, but for speed and scale, the new embedding models have proven they can match the accuracy of the giants while running a thousand times faster.

Technical Summary: Frontier LLMs vs. Natively Multimodal Embeddings for Hard-Negative Text-to-Image Retrieval

Problem Statement

Text-to-image retrieval, the task of ranking images by relevance to a natural language query, has traditionally relied on dual-encoder models (e.g., CLIP) that align visual and textual representations via contrastive learning. Two recent developments challenge this paradigm:

Frontier Large Language Models (LLMs): Models like GPT-4.1 and Claude Sonnet 4.6 possess strong visual capabilities, enabling a new retrieval paradigm where all candidate images and the query are presented in a single prompt for direct zero-shot ranking.
Natively Multimodal Embeddings: Models such as Google's Gemini Embedding 2 (GE2) and Amazon Nova 2 project text, images, video, and audio into a single shared embedding space, moving beyond the dual-encoder architecture.

While both approaches show promise, their relative performance on retrieval accuracy and ranking latency—specifically under challenging conditions—remains uncharacterized in the literature. This study addresses the gap by comparing native multimodal embeddings against frontier LLMs as zero-shot direct visual rankers.

Methodology

Dataset and Candidate Construction

The study utilizes Flickr30k (Young et al., 2014), a dataset of 31,000 images with five independent captions each. To create a non-trivial evaluation task, the authors constructed hard-negative candidate sets:

Pool: 5,000 images were sampled, each with five captions.
Query Selection: 1,000 captions were randomly selected as queries.
Hard Negatives: For each query, a candidate set of 25 images was formed: the ground-truth image plus 24 "hard negatives."
Selection Logic: Hard negatives were selected based on caption-space similarity rather than image-space similarity. Using sentence-transformers/all-mpnet-base-v2, the 24 captions most semantically similar to the query caption (excluding the query itself) were identified. The images corresponding to these captions formed the negative set. This approach increases ambiguity, as images are difficult to distinguish given the query caption.

Evaluated Systems

Four systems were evaluated on the same 1,000 queries and identical candidate sets:

Gemini Embedding 2 (GE2): A natively multimodal embedding model. Queries and images were embedded separately using specific task types (RETRIEVAL_QUERY and RETRIEVAL_DOCUMENT), and candidates were ranked by cosine similarity.
Amazon Nova 2: A multimodal embedding model. Similar to GE2, but used GENERIC_INDEX for embedding purposes.
GPT-4.1: A frontier LLM. The query caption and all 25 candidate images were presented in a single zero-shot prompt. The model was instructed to return a ranked list of image IDs.
Claude Sonnet 4.6: A frontier LLM. Evaluated using the same prompt and procedure as GPT-4.1.

To mitigate position bias in LLMs, the order of images in the prompt was randomized for each query.

Evaluation Metrics

Performance was measured using:

Recall@1 (R@1): Fraction of queries where the ground-truth image is ranked first.
Recall@3 (R@3): Fraction of queries where the ground-truth image is in the top three.
Mean Reciprocal Rank (MRR): Mean of the reciprocal rank of the ground-truth image.
Latency: Execution time for ranking 1,000 queries, distinguishing between end-to-end processing and scenarios with precomputed embeddings.

Statistical significance was assessed using McNemar's test for R@1 and the Wilcoxon signed-rank test for MRR, with Bonferroni correction.

Key Results

Retrieval Accuracy

Top Performers: Gemini Embedding 2, GPT-4.1, and Claude Sonnet 4.6 achieved statistically indistinguishable performance across all three metrics (R@1, R@3, MRR).
- R@1 Scores: GE2 (0.804), Claude Sonnet 4.6 (0.796), GPT-4.1 (0.789).
- Significance: Pairwise tests showed no significant difference between these three systems (all $p > 0.09$ ).
Underperformer: Amazon Nova 2 trailed significantly, achieving an R@1 of 0.673. Statistical tests confirmed its performance was significantly different from all other systems ( $p < 0.0001$ ). The authors attribute this gap to the use of GENERIC_INDEX for embedding purposes, which is optimized for general vector database creation rather than specific text-image matching.

Latency and Efficiency

Precomputation Advantage: Embedding models (GE2 and Nova 2) allow for precomputation of image embeddings. Once precomputed, ranking 1,000 queries with 25 candidates each took under 2 seconds (0.66s for GE2, 1.16s for Nova 2).
LLM Latency: LLMs process queries end-to-end without precomputation. Ranking the same workload took 6,100 seconds for GPT-4.1 and 9,415 seconds for Claude Sonnet 4.6.
Conclusion: Embedding models offer a speedup of approximately four orders of magnitude for low-latency applications involving fixed image galleries.

Contributions and Significance

The paper claims the following contributions:

First Direct Comparison: It provides the first direct comparison of native multimodal embeddings against frontier LLMs as zero-shot direct visual rankers on a hard-negative dataset (Flickr30k), without task-specific fine-tuning or intermediate caption generation.
Accuracy Parity: The study empirically demonstrates that dedicated multimodal embedding models (specifically GE2) and frontier LLMs (GPT-4.1, Claude Sonnet 4.6) achieve statistically indistinguishable retrieval accuracy on hard-negative candidates. This suggests that LLMs can effectively serve as zero-shot rankers, matching the performance of specialized embedding spaces.
Latency Trade-offs: The paper highlights a critical practical distinction: while accuracy is comparable, embedding models are vastly superior for low-latency applications due to precomputation capabilities. Conversely, LLMs may be preferable for frequently updated galleries or small sets where precomputation is impractical.

Limitations

The authors acknowledge several limitations:

Ambiguous Ground Truth: In hard-negative sets, a caption from one image may plausibly describe another, creating ambiguity. However, this affects all systems equally.
Caption-Space Hardness: Hard negatives were defined in caption space, which may inherently favor embedding-based methods optimized for dense vector representations.
Inference Conditions: LLMs benefit from cross-candidate reasoning (seeing all 25 images at once), whereas embedding models score images independently.
Proprietary Access: All systems are accessed via paid APIs, and model weights may change between experiments and replication attempts.
Scope: The evaluation uses 25-image candidate sets, measuring ranking performance rather than full-gallery retrieval, which may not generalize to massive-scale settings.

Conclusion

The study concludes that frontier LLMs can match the retrieval accuracy of native multimodal embedding models on hard-negative text-to-image retrieval tasks. However, the choice of system depends on the application constraints: embedding models are the superior choice for low-latency, large-scale retrieval with fixed galleries, while LLMs offer a viable zero-shot alternative for dynamic or smaller-scale scenarios where precomputation is not feasible.