💻 computer science

FreeRet: MLLMs as Training-Free Retrievers

FreeRet is a training-free, plug-and-play framework that leverages off-the-shelf Multimodal Large Language Models (MLLMs) as powerful two-stage retrievers by generating semantically grounded embeddings for candidate search and utilizing their inherent reasoning capabilities for precise reranking, thereby outperforming heavily trained models without requiring additional fine-tuning.

Original authors: Yuhan Zhu, Xiangyu Zeng, Chenting Wang, Xinhao Li, Chunxu Liu, Yicheng Xu, Ziang Yan, Yi Wang, Limin Wang

Published 2026-05-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuhan Zhu, Xiangyu Zeng, Chenting Wang, Xinhao Li, Chunxu Liu, Yicheng Xu, Ziang Yan, Yi Wang, Limin Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, well-read librarian who has read millions of books and seen countless pictures. This librarian is incredibly smart, can understand complex stories, and can even write new ones. However, traditionally, if you wanted this librarian to act as a search engine (finding specific items based on a query), you had to put them through years of grueling, expensive retraining. You'd have to force them to memorize specific "search rules" and ignore their natural ability to just "chat" and "think."

The paper FreeRet asks a simple question: What if we don't need to retrain the librarian at all? What if we can just ask them to do the search job using their existing brainpower?

Here is how they did it, explained through three simple steps:

1. The "Final Filter" Problem (Skipping the Lexicalization Layer)

The Analogy: Imagine the librarian is trying to summarize a complex book. They understand the deep meaning perfectly. But, right before they speak, they have to pass their thoughts through a "translator" that forces them to use only simple, common words (like "cat," "run," "happy") to match a dictionary. This translator is great for speaking, but it ruins the deep, nuanced meaning of the book.

The Fix: The paper found that the last part of the AI's brain (the "final MLP layer") acts like this translator. It forces the AI to focus on simple word matching rather than deep meaning.

FreeRet's Trick: They simply tell the AI to skip this final translator. They grab the "thought" before it gets forced into simple words. This gives them a much richer, more accurate "summary" of the image or text, making the initial search much smarter without changing a single line of code.

2. The "Vague Summary" Problem (Controlled Generation)

The Analogy: If you ask a smart person, "Summarize this picture in one word," they might say "Thing" or "Stuff." It's technically a word, but it's useless for finding the right picture later. Or, they might say "Growing" (focusing on the wrong part of the image).

The Fix: The paper realized that just asking for "one word" is too loose.

FreeRet's Trick: They give the AI a strict set of instructions (a "prompt") before it speaks. They say: "Look at the image and the text. Capture the main meaning. Do not use small words like 'the' or 'is'. Focus on the topic."
By giving these specific rules, the AI generates a much more precise "one-word" summary that actually helps find the right match.

3. The "Framing Effect" Problem (The Reranking Trick)

The Analogy: Imagine you are a judge deciding if a suspect is guilty.

If you ask, "Is the suspect Right or Wrong?" you might feel a moral pressure to say "Right" because it sounds like a moral judgment.
If you ask, "Is the suspect True or False?" you might feel more neutral.
The paper found that the AI's answer changes depending on how you ask the question, even if the meaning is the same. This is called the "Framing Effect."

The Fix: The paper discovered that asking the AI to simply say "Yes" or "No" introduces bias.

FreeRet's Trick: They turn the question into a Multiple Choice Question (MCQ). Instead of asking "Is it relevant?", they ask: "Does the candidate match the query? A. Yes, it matches. B. No, it doesn't."
This format is something the AI has seen millions of times in its training data. It makes the AI act like a neutral judge, ignoring the emotional weight of words like "Yes" or "Wrong," leading to much more accurate final results.

The Result: A "Plug-and-Play" Super-Search Engine

The paper tested this on a massive benchmark called MMEB (which has 36 different types of search tasks, from finding images to answering video questions).

The Surprise: Their method, FreeRet, which uses zero training data and zero extra cost, beat models that were trained on millions of examples.
The Benefit: Because they didn't retrain the model, the AI keeps all its original superpowers. It can still chat, write stories, and reason about complex topics. You don't have to choose between a "search bot" and a "smart assistant"; FreeRet makes the same model do both jobs perfectly.

In short: FreeRet shows that we don't need to force smart AI models to learn how to search. We just need to ask them the right questions and let them use their existing, pre-trained brains to do the job.

1. Problem Statement

Multimodal Large Language Models (MLLMs) possess powerful reasoning and cross-modal understanding capabilities, yet they are rarely used directly for retrieval tasks. Current state-of-the-art approaches rely on heavy post-hoc fine-tuning (contrastive learning) to convert MLLMs into retrievers. This paradigm faces three critical limitations:

High Cost: It requires massive amounts of paired multimodal data and expensive computational resources for every new backbone or modality configuration.
Poor Generalization: Models fine-tuned on specific benchmarks (e.g., MMEB) often fail to generalize to other domains or modality combinations without re-training.
Fragmented Architecture: Existing solutions often separate the embedding (search) and reranking stages into different modules or require auxiliary models, breaking the end-to-end nature of the MLLM.

The authors ask: Can off-the-shelf MLLMs serve as powerful, two-stage retrievers without any additional training?

2. Methodology: The FreeRet Framework

FreeRet is a plug-and-play, training-free framework that transforms any pre-trained MLLM into a two-stage retriever (Embedding $\rightarrow$ Reranking) by leveraging three core technical innovations:

A. Bypassing Lexicalization Pressure (Embedding Extraction)

The authors identify that the final Multi-Layer Perceptron (MLP) layer in MLLMs exerts "lexicalization pressure," forcing semantic representations to align with discrete token prediction rather than preserving continuous semantic structure.

Technique: Instead of using the standard final hidden state ( $h^L_{MLP}$ ), FreeRet extracts embeddings from the output of the Attention sub-layer ( $h^L_{Attn}$ ) of the final transformer layer.
Effect: This bypasses the projection into the vocabulary space, retaining high-level semantic abstractions and preventing the distortion of semantic continuity caused by the need to predict specific tokens.

B. Controlled Embedding Generation

Standard training-free methods often use simple prompts like "Summarize in one word," leading to vague or noisy outputs (e.g., function words).

Technique: FreeRet employs a controlled generation prompt with three constraints:
1. Task Alignment: Explicitly instructs the model to assess relevance to a specific topic.
2. Semantic Grounding: Anchors the summary to the input content.
3. Noise Suppression: Explicitly forbids function words, prepositions, or symbols.
Effect: This steers the model to generate semantically faithful, task-relevant tokens, resulting in higher-quality embeddings without parameter updates.

C. Mitigating the Framing Effect (Reranking)

The authors discovered a "Framing Effect" in MLLMs where logically equivalent binary labels (e.g., Yes/No vs. True/False) yield significantly different accuracy due to pre-training biases and pragmatic connotations.

Technique: Instead of binary classification prompts, FreeRet frames the reranking task as a Multiple-Choice Question (MCQ):
- Option A: "Yes, the candidate fully matches the query."
- Option B: "No, the candidate does not match."
Effect: The MCQ format neutralizes semantic biases and aligns with the distribution of pre-training data, leading to more stable and accurate relevance scoring.

3. Key Contributions

Training-Free Paradigm: Demonstrates that MLLMs can achieve state-of-the-art retrieval performance without any fine-tuning, auxiliary models, or external data.
Architectural Insight: Identifies and solves the "lexicalization pressure" problem by removing the final MLP layer for embedding extraction, a finding validated across model scales (3B to 32B).
Unified RAG Pipeline: Unifies embedding, reranking, and generation into a single model. This preserves the MLLM's generative and conversational abilities while adding retrieval capabilities.
Scalability: The framework is model-agnostic and supports arbitrary modality combinations (e.g., audio-to-video, image+text-to-video) by leveraging the intrinsic omni-modal understanding of the base model.

4. Experimental Results

The framework was evaluated on MMEB (36 datasets across 4 meta-tasks) and MMEB-V2 (video subset).

Performance vs. Supervised Models:
- FreeRet-7B (using Qwen2.5-VL-7B) achieves an average Precision@1 of 70.7% on MMEB.
- It outperforms models trained on millions of pairs, such as GME-7B (trained on 8M pairs) and MMRet-7B (trained on 26.2M pairs).
- It matches or exceeds methods explicitly supervised on MMEB (e.g., UniME-V2).
Video Retrieval: On MMEB-V2, FreeRet significantly surpasses supervised baselines in both video classification and retrieval tasks.
Efficiency: By using a single model for both stages and reusing KV caches, FreeRet reduces memory footprint and latency compared to two-stage systems using separate embedding and reranking models.
Generalization: The method works effectively across different model families (LLaVA, InternVL, Qwen) and scales (2B to 32B) without re-tuning.

5. Significance and Impact

Democratization of Retrieval: Removes the barrier of expensive data curation and fine-tuning, allowing immediate deployment of retrieval capabilities on any pre-trained MLLM.
Paradigm Shift: Challenges the necessity of large-scale supervised contrastive training for multimodal retrieval, suggesting that the reasoning and representation capabilities of foundation models are sufficient when properly harnessed.
Long-Video Understanding: Demonstrates that retrieval-driven focus (selecting relevant frames before reasoning) significantly improves performance on long-video benchmarks (LVBench), offering a scalable solution for temporal reasoning.
Future Direction: Points toward a future where a single "generalist" model unifies retrieval, reranking, and generation, simplifying the architecture of complex AI systems.

In conclusion, FreeRet proves that with careful prompt engineering and architectural adjustments (bypassing the final MLP), pre-trained MLLMs can serve as robust, versatile, and training-free engines for multimodal retrieval.