Reason to Contrast: A Cascaded Multimodal Retrieval Framework

1. Problem Statement

Traditional multimodal retrieval systems rely heavily on bi-encoder architectures, where performance is constrained by embedding dimensionality and model size. While recent work like Think-Then-Embed (TTE) improved retrieval by generating reasoning tokens (Embedding-Centric Reasoning, or ECR) before embedding, it still faces limitations:

• Lack of Joint Reasoning: TTE generates reasoning for the query and candidate independently. It lacks the ability to perform joint reasoning between query-candidate pairs, which is crucial for fine-grained ranking in complex tasks (e.g., video retrieval where the target contains significantly more information than the query).
• Coarse Embeddings: Even with pre-embedding reasoning, the initial embeddings often fail to capture the subtle distinctions needed to rank the most relevant item at the very top (Top-1), especially in high-information-density tasks like video retrieval.
• Inefficient Scaling: Scaling performance traditionally requires larger models or more training data. There is a need for a scaling axis based on test-time reasoning tokens rather than model parameters.
• Suboptimal Hard Negative Mining: Existing methods for Hard Negative Mining (HNM) often rely on the embedder itself or simple heuristics, which can be biased or inaccurate compared to a stronger, unbiased reasoning model.

2. Methodology: TTE-v2 Framework

The authors propose TTE-v2, a cascaded multimodal retrieval framework that introduces a "Reason-to-Contrast" paradigm. It operates in two main stages: Retrieval and Reranking, with a feedback loop for training.

A. Core Components

1. ECR-based Reranking (ECRR):

   • Instead of performing expensive multimodal reranking (query + image/video), TTE-v2 reuses the Embedding-Centric Reasoning (ECR) traces generated during the initial retrieval stage.
   • Since ECRs are text, the reranker performs text-only reranking (pairwise or listwise) using an MLLM. This drastically reduces computational cost while maintaining high expressiveness.
   • Pairwise: The reranker scores each candidate's ECR against the query.
   • Listwise: The reranker processes the query and a list of candidate ECRs together to output a ranked list.

2. Query-Aware Reasoning (QAR):

   • Standard ECRs can be too general and may miss query-specific details.
   • QAR introduces a rewriting step where the MLLM rewrites the candidate's ECR to explicitly incorporate the query context.
   • This creates a Query-Aware ECR that highlights discriminative information relevant to the specific query, enabling more precise contrastive reasoning during reranking.

3. Hard Negative Mining with Reranking Feedback (rHNM):

   • The authors introduce a feedback loop where the reranker acts as a teacher for the retriever.
   • Process:
     1. A pretrained embedder retrieves a coarse pool of candidates.
     2. The ECRR module reranks these candidates.
     3. False Negative Filtering: Candidates ranked higher than the ground truth by the strong reranker are identified as potential false negatives and removed.
     4. Hard Negative Selection: The remaining candidates are used as hard negatives, weighted by their reranking scores.
   • This allows the system to distill the reranker's superior joint reasoning capabilities back into the embedding model during training.

B. Inference Pipeline

1. Stage 1 (Retrieve): Query $\to$ ECR Generation $\to$ Embedding $\to$ Retrieve Top-$K$ candidates (and their ECRs).
2. Stage 2 (Rerank):
   • (Optional) QAR: Rewrite candidate ECRs based on the query.
   • ECRR: Perform text-only reranking using the (rewritten) ECRs to produce the final ranking.

3. Key Contributions

1. TTE-v2 Framework: A novel cascaded retrieval-reranking architecture that shifts the scaling paradigm from model size/embedding dimension to test-time reasoning token budget.
2. Query-Aware Reasoning (QAR): A mechanism to refine candidate reasoning traces to be query-specific, significantly improving discriminative power for complex tasks.
3. ECR-based Reranking (ECRR): A lightweight, text-only reranking strategy that leverages pre-generated reasoning traces, offering a cost-effective alternative to heavy multimodal reranking.
4. rHNM (Reranker-based Hard Negative Mining): A strategy using the reranker as an unbiased teacher to filter false negatives and mine harder negatives, effectively distilling knowledge back into the retriever.
5. State-of-the-Art Performance: Establishing new SOTA on the MMEB-V2 benchmark without relying on additional private training data.

4. Experimental Results

The paper evaluates TTE-v2 on the MMEB-V2 benchmark (78 tasks across image, video, and visual documents) and out-of-distribution datasets.

• Overall Performance:
  • TTE-v2-7B achieves 75.7% overall accuracy, surpassing the previous SOTA (IFM-TTE-7B at 75.1%) and proprietary models trained on massive private datasets.
  • TTE-v2-2B (2B parameters) matches or surpasses leading 7B models trained with significantly larger external data, demonstrating the efficiency of the token-wise scaling approach.
• Video Retrieval:
  • TTE-v2 shows substantial gains in video tasks (e.g., +6% improvement over previous bests).
  • The 2B variant improves over the previous best 7B model by +5.4% on average.
• Ablation Studies:
  • ECRR: Adds ~10% absolute improvement over the TTE baseline.
  • QAR: Further boosts performance, particularly on tasks requiring complex joint reasoning (e.g., QVHighlight, Charades-STA).
  • rHNM: Contributes an additional +1% overall gain and significantly reduces the false negative ratio compared to embedder-based mining.
• Efficiency: The text-only reranking (ECRR) is significantly cheaper than multimodal reranking while outperforming zero-shot MLLM rerankers that struggle with multiple video inputs.

5. Significance

• New Scaling Axis: The paper challenges the notion that retrieval performance must scale with model parameters or embedding dimensions. It proves that scaling reasoning tokens at test time is a viable and powerful alternative.
• Coarse-to-Fine Paradigm: It successfully bridges the gap between fast, coarse retrieval and slow, fine-grained reasoning, making high-performance multimodal retrieval more accessible and efficient.
• Feedback Loop: The rHNM mechanism demonstrates how a strong, expensive reranker can be used to "teach" a smaller, faster retriever, creating a self-improving system.
• Practical Impact: By achieving SOTA with smaller models (2B) and without massive private datasets, TTE-v2 offers a highly practical solution for real-world multimodal search applications where cost and latency are critical.