Technical Summary: UniRank

Problem Statement

Multimodal information retrieval (IR) systems increasingly face the challenge of hybrid candidate sets, where the pool of items to be reranked contains a mix of text-only and image-only documents. While text-only rerankers are mature, extending them to multimodal settings introduces a modality gap: models inherently biased toward text struggle to fairly compare relevance across heterogeneous modalities.

Existing solutions suffer from two primary limitations:

1. Modality Conversion Overhead: Common approaches convert images to text (via captions or OCR) or text to images to force a single input format. Converting text to images (as done by some Vision-Language Model (VLM) rerankers) incurs substantial storage and inference overhead, while converting images to text often loses fine-grained visual semantics.
2. Domain Mismatch: Most open-source multimodal rerankers are trained on general web-scale data. They often underperform in specific domains (e.g., scientific literature, design patents) where private data and unique semantic structures dominate.

There is a lack of comprehensive, end-to-end frameworks capable of natively scoring hybrid text-image candidates without modality conversion while adapting to specific domains.

Methodology: UniRank Framework

The authors propose UniRank, a VLM-based reranking framework designed to score and order hybrid text-image candidates natively. The framework employs a two-stage, end-to-end domain adaptation pipeline:

1. Problem Formulation & Interface

UniRank treats reranking as an instruction-following task. Given a query $q$ and a candidate $c$ (which may be text, image, or both), the VLM is prompted to output a discrete relevance label (e.g., "yes" or "no").

• Native Scoring: The model processes inputs in their original modalities.
• Score Derivation: The scalar relevance score $s_\theta(q, c)$ is derived from the logit difference between the "yes" and "no" tokens:
  $$s_\theta(q, c) = \sigma(\text{logit}_\theta(\text{yes} | q, c) - \text{logit}_\theta(\text{no} | q, c))$$
  This allows a unified scoring function for text-text, image-image, and cross-modal pairs.

2. Stage 1: Instruction-Driven Supervised Fine-Tuning (SFT)

To adapt a general-purpose VLM to the reranking task:

• Data Construction: Labeled query-candidate pairs are serialized into chat-style instructions. The system prompt restricts the output to a single label, while the user prompt interleaves the query and candidate in their native formats.
• Objective: The model is fine-tuned using standard language modeling loss on the assistant response tokens. This stage teaches the model to produce calibrated cross-modal relevance scores specific to the target domain.
• Hold-out Split: A subset of data is reserved (not used in SFT) specifically for constructing preference data in the next stage to prevent overfitting.

3. Stage 2: Hard-Negative Driven Preference Alignment (RLHF)

To further refine ranking robustness and calibration:

• Hard-Negative Mining: The SFT model is run on the hold-out data to identify "hard negatives"—candidates that the model scores highly but are actually non-relevant.
• Preference Construction: Pairwise preference data is constructed by contrasting the ground-truth label against the model's error mode (e.g., Chosen: "no" vs. Rejected: "yes" for a hard negative).
• Reward Model (RM) Training: A reward model is trained on these pairwise preferences using a Bradley-Terry style loss to predict which response is preferred.
• Policy Optimization (RLHF): The reranker (policy) is optimized using Group Relative Policy Optimization (GRPO).
  • Key Innovation: Instead of standard prompt-level grouping, UniRank uses query-level grouping. It contrasts multiple candidates sharing the same query within a group. This directly optimizes the relative ordering behavior required for reranking, rather than just individual relevance judgments.

Key Contributions

1. Hybrid-Modality VLM Reranking: UniRank is the first framework to natively score hybrid text-image candidates without converting text to images, thereby eliminating the storage and inference overhead associated with image-homogenization strategies.
2. End-to-End Domain-Specific Alignment: The paper proposes a complete training pipeline comprising:
   • Instruction-driven SFT for calibrated scoring.
   • Hard-negative mining to capture in-domain failure modes.
   • RLHF with query-level grouping to optimize ranking robustness.
3. Empirical Validation: The framework is validated on two distinct real-world domains: scientific literature retrieval and design patent search.

Experimental Results

The authors evaluated UniRank on the MMDocIR benchmark (Scientific Literature) and a custom Design Patent Search dataset.

• Performance: UniRank consistently outperforms state-of-the-art baselines, including strong text-only rerankers (e.g., BGE-Reranker, Qwen3-Reranker) and existing VLM-based rerankers (e.g., MM-R5, Qwen3-VL-Reranker).
  • Scientific Literature: Improved Recall@1 by 8.9% over the best baseline.
  • Design Patent: Improved Recall@1 by 7.3% over the best baseline.
• Efficiency: By avoiding text-to-image conversion, UniRank significantly reduces storage requirements (up to 6.9$\times$ less in the scientific domain) and inference latency (up to 2.8$\times$ faster) compared to image-homogenizing baselines like MM-R5.
• Robustness: The model maintains performance gains across different first-stage retrievers (text-based and visual-based) and scales effectively with backbone model sizes (from 2B to 8B parameters).
• Ablation Studies:
  • Removing SFT causes the largest performance drop, confirming its necessity for establishing a calibrated scoring interface.
  • Hard-negative mining is critical; replacing it with random negatives degrades performance.
  • Query-level grouping in RLHF outperforms standard prompt-level grouping, validating the importance of optimizing relative ordering.

Significance and Claims

The paper claims that UniRank addresses a critical gap in multimodal IR by providing a practical, efficient, and domain-adaptive solution for hybrid candidate sets.

• Modality Handling: It demonstrates that native hybrid scoring is superior to modality conversion, offering both higher accuracy and lower computational costs.
• Domain Adaptation: It highlights that general-domain VLMs are insufficient for specialized tasks and that a structured, two-stage alignment process (SFT + RLHF with hard negatives) is essential for high-performance deployment in privacy-sensitive or domain-specific scenarios.
• Scalability: The framework is presented as a scalable recipe for building retrieval systems where candidate sets are inherently heterogeneous, bridging the gap between theoretical VLM capabilities and practical IR system requirements.

The authors position UniRank not merely as a model improvement but as a framework that enables the deployment of high-performance multimodal retrieval in real-world applications where data privacy, efficiency, and domain specificity are paramount.