BRIDGE: Multimodal-to-Text Retrieval via Reinforcement-Learned Query Alignment

1. Problem Statement

The paper addresses the significant performance gap in Multimodal-to-Text Retrieval, where systems attempt to retrieve relevant text documents from a corpus using queries that contain both text and images.

• The Bottleneck: Current state-of-the-art multimodal retrievers (using Vision-Language Models like CLIP or Nomic-Vision) underperform compared to strong text-only retrievers. On the MM-BRIGHT benchmark, the best multimodal model achieves only 27.6 nDCG@10, whereas the best text-only retriever achieves 32.2 nDCG@10.
• Root Cause Analysis: The authors argue that the failure is not due to a lack of visual understanding in the retriever, but rather query representation noise. Raw multimodal queries often entangle:
  • Visual descriptions (captions).
  • Conversational context (e.g., "Why is my service failing?").
  • Task-irrelevant background.
  • The actual retrieval intent.
    This "entanglement" creates noisy embeddings that confuse dense retrieval models, causing them to fail at identifying the core intent required to match text documents.

2. Methodology: The BRIDGE Framework

BRIDGE proposes a two-component system that resolves this mismatch by transforming the query before retrieval, operating entirely in the text modality during inference (no visual encoder is needed at retrieval time).

Component A: FORGE (Focused Retrieval Query Generator)

• Function: A query alignment model that distills noisy multimodal inputs into compact, retrieval-optimized search strings.
• Architecture: Built on Qwen2.5-7B-Instruct.
• Training Strategy (Reinforcement Learning):
  • Unlike supervised approaches that mimic human rewrites, FORGE is trained via Group Relative Policy Optimization (GRPO).
  • Input: A raw multimodal query $(q_t, q_v)$ where the image $q_v$ is first converted to a dense text caption $\delta(q_v)$ using GPT-4o.
  • Action: The model generates a concise search string $\hat{q}$ (max 200 words).
  • Reward Signal: The reward is defined by the downstream retrieval performance. Specifically, the model is rewarded if the generated query $\hat{q}$ causes the retriever (LENS) to rank the ground-truth document highly (measured by nDCG@k).
  • Goal: To learn how to strip conversational noise and absorb visual context into a single, intent-focused text query.

Component B: LENS (Language-Enhanced Neural Search)

• Function: A reasoning-enhanced dense retriever designed to handle the structured, intent-rich queries produced by FORGE.
• Architecture: A bi-encoder based on Qwen3-Embedding-4B.
• Training: Fine-tuned on reasoning-intensive retrieval data (math, science, law, software) using contrastive learning with hard negatives. It encodes the aligned query $\hat{q}$ and documents into a shared vector space for cosine similarity search.

Inference Pipeline

1. Visual Captioning: GPT-4o converts the query image into a text description.
2. Query Alignment: FORGE takes the text question + image description and outputs a distilled search string.
3. Retrieval: LENS encodes the distilled string and retrieves the top-$k$ documents from the text corpus.

3. Key Contributions

1. Reframing the Problem: Identifies that the bottleneck in multimodal retrieval is query representation noise, not model capacity or visual encoding limitations.
2. FORGE (RL-based Alignment): Introduces a reinforcement-learned query generator that optimizes directly for retrieval quality rather than mimicking reference rewrites. It eliminates the need for multimodal encoders at inference time.
3. LENS: A specialized dense retriever fine-tuned to handle the specific type of intent-rich queries generated by FORGE.
4. Plug-and-Play Capability: Demonstrates that FORGE can be applied as a pre-processing layer to any existing retriever (including multimodal ones) to boost performance.

4. Experimental Results

Evaluated on MM-BRIGHT (2,803 queries across 29 technical domains):

• Main Performance:
  • BRIDGE (FORGE + LENS): Achieves 29.7 nDCG@10.
  • Comparison: Surpasses the strongest multimodal baseline, Nomic-Vision (27.6), by +2.1 points.
  • Gap Reduction: Reduces the performance gap between multimodal and text-only retrieval by 46% (from 4.6 points down to 2.5 points).
• Plug-and-Play Effect:
  • When FORGE is applied as a pre-processor to Nomic-Vision, the combined system reaches 33.3 nDCG@10, exceeding the best text-only retriever (32.2). This proves that query alignment is the missing ingredient, regardless of the underlying retriever.
• Ablation Studies:
  • RL vs. Supervised: The RL-trained FORGE (29.7) significantly outperforms a supervised version (28.5) and raw concatenation (27.8), proving that optimizing for retrieval rewards is superior to mimicking human rewrites.
  • Backbone Scaling: Interestingly, a 11B parameter model (Llama-3.2) performed slightly better (31.0) than the 7B default or larger 72B models, suggesting that instruction-following quality and conciseness matter more than raw parameter count for this specific task.
  • Comparison to Rewriting Baselines: FORGE vastly outperforms standard query rewriting methods like HyDE and Query2Doc, which tend to amplify noise in multimodal settings.

5. Significance and Impact

• Efficiency: BRIDGE offers a lightweight, modular solution. By removing the need for heavy multimodal encoders at inference time, it is more scalable and cost-effective than current VLM-based retrieval systems.
• Paradigm Shift: The paper challenges the prevailing assumption that better visual encoders are the solution to multimodal retrieval. Instead, it posits that cleaning the query is the critical step.
• Generalizability: The "Plug-and-Play" nature of FORGE suggests that this alignment strategy can be retrofitted onto existing retrieval pipelines to immediately boost performance without retraining the entire retrieval model.
• Future Direction: The results suggest that the remaining gap between multimodal and text-only retrieval is purely a representation issue, implying that with further query alignment optimization, multimodal retrieval could potentially match or exceed text-only performance.