Efficient Discriminative Joint Encoders for Large Scale Vision-Language Reranking

This paper introduces EDJE, an efficient discriminative joint encoder that precomputes and compresses visual tokens to overcome the high computational and storage costs of existing vision-language rerankers, enabling high-throughput inference with minimal disk usage while maintaining state-of-the-art retrieval performance.

The Gist

The Big Problem: The "Slow Chef" in a Fast Food Kitchen

Imagine you run a massive library (a database of millions of images) and people are asking for books (images) based on descriptions (text).

1. The Old Way (Embedding Models): You have a super-fast librarian who can quickly scan the spine of every book and guess what's inside. This is fast, but sometimes the guess is wrong because they didn't read the actual pages.
2. The "Perfect" Way (Joint Encoders): You have a genius scholar who reads the entire book and the entire description, then compares them word-for-word. This is incredibly accurate, but it takes forever. If you have to do this for 50,000 books, your customer waits hours.

The Bottleneck: The paper points out that the "genius scholar" (like the famous BLIP model) is too slow because they spend 90% of their time just looking at the pictures to understand them before they even start reading the text. It's like a chef spending 45 minutes chopping vegetables before they even start cooking the meal.

The Solution: EDJE (The "Pre-Cooked" Chef)

The authors introduce EDJE (Efficient Discriminative Joint Encoder). Their big idea is to change when the chef does the hard work.

1. The "Pre-Cooked" Strategy (Offline Precomputation)

Instead of chopping vegetables (extracting image features) every time a customer orders, EDJE says: "Let's chop all the vegetables once, store them in the fridge, and just grab them when needed."

• How it works: They take all the images in the database once, process them into a compact "summary" (tokens), and save them on the hard drive.
• The Benefit: When a user types a query, the system doesn't need to "look" at the raw image again. It just grabs the pre-made summary. This saves a massive amount of time.

2. The "Condensed Summary" (Token Compression)

Here is the tricky part. Even the "summary" of an image is huge. If you have 1 million images, and each summary is 100MB, your hard drive will explode.

• The Metaphor: Imagine the image summary is a 500-page novel. Storing 1 million novels is impossible.
• The Fix: EDJE uses a special Adapter (a smart filter). It reads the 500-page novel and writes a 64-word summary that captures the most important parts.
• The Magic: This 64-word summary is tiny (only 49 kilobytes per image!), but it still contains the "soul" of the image. It's like turning a whole movie into a perfect 1-minute trailer that still tells you exactly what the movie is about.

3. The "Fast Matchmaker" (The Joint Encoder)

Now, when a user asks, "Show me a picture of a dog playing in the snow," the system:

1. Takes the text.
2. Grabs the tiny 64-word summaries of the top 50,000 candidate images from the fridge.
3. Feeds the text and the summaries into a small, fast language model (like a MiniLM).
4. This model acts as a matchmaker, comparing the text to the summaries instantly.

Because the heavy lifting (looking at the raw image) was done offline, and the data is tiny, this matchmaker can process 50,000 pairs per second. That's like checking 50,000 books in the time it takes to blink.

Why This Matters (The Results)

• Speed: It is up to 53 times faster than previous "perfect" models.
• Storage: It uses 49KB of space per image (compressed) instead of megabytes. You could store millions of these on a standard laptop.
• Accuracy: Even though it's fast and uses tiny summaries, it is just as good at finding the right picture as the slow, heavy models. In fact, it beats the old "fast but dumb" models significantly.

A Real-World Analogy: The Dating App

• Old Fast Models (CLIP): You swipe through photos based on a quick glance. "Looks like a dog." Fast, but you might miss the specific breed or the fact that the dog is wearing a hat.
• Old Slow Models (BLIP): You read a full biography of every person before deciding to swipe. Accurate, but you'd never find a date because it takes too long.
• EDJE: You have a database of "Pre-written Bios" (the compressed tokens). When you search for "Dog with a hat," the system instantly matches your text against these pre-written bios. It's fast enough to handle millions of users, but smart enough to know the difference between a dog in a hat and a dog in a tuxedo.

Summary

The paper solves the problem of "How do we make AI look at pictures and read text together really fast?" by saying: "Don't look at the pictures in real-time. Summarize them beforehand, shrink the summaries down to their absolute essentials, and then match them quickly."

This allows us to build massive, smart image search engines that are both incredibly accurate and lightning-fast.

Technical Summary

1. Problem Statement

Large-scale multimodal retrieval (finding images for text queries or vice versa) currently relies heavily on embedding-based models (e.g., CLIP, SigLIP) for fast vector search. While these models are efficient, they suffer from "late interaction," meaning they process image and text modalities independently before comparing them, limiting fine-grained cross-modal understanding.

In contrast, joint encoders (e.g., BLIP, BLIP-2) process both modalities simultaneously, allowing for deep cross-attention and significantly better re-ranking performance. However, these models have not been adopted in large-scale retrieval pipelines due to a critical bottleneck:

• Expensive Online Computation: Existing joint encoders require heavy, high-resolution Vision Transformers (ViT) to extract visual features online for every query. This extraction consumes the vast majority of inference time (up to 93% in some cases), making it infeasible to re-rank thousands of candidates per query in real-time.
• Storage Costs: Storing full visual token sequences (e.g., 576 tokens per image) for billions of images is prohibitively expensive in terms of disk space.

The Core Question: Can we harness the discriminative power of joint encoders for large-scale retrieval while maintaining the efficiency required for high-throughput inference?

2. Methodology: EDJE Architecture

The authors propose EDJE (Efficient Discriminative Joint Encoder), a system that shifts the computational burden of vision feature extraction to an offline stage and compresses the data for online use.

A. Offline Vision Precomputation

Instead of extracting features online, EDJE treats the vision encoder as a preprocessing stage:

1. Images are encoded once by a powerful vision backbone (e.g., SigLIP2).
2. The resulting visual tokens are stored on disk.
3. This allows the use of massive, high-resolution vision encoders without slowing down the online retrieval process.

B. Token-Compression Adapter

Storing full token sequences (e.g., 576 tokens) is still too storage-intensive. EDJE introduces a lightweight attention-based adapter to compress these tokens:

• Mechanism: A set of $m$ learnable "universal query tokens" ($Q$) performs cross-attention over the $n$ visual tokens ($X$) extracted by the vision encoder.
• Process:
  1. $K = XW_K, V = XW_V$ (Linear projections of visual tokens).
  2. $H = \text{MultiHeadAttention}(Q, K, V)$ (Aggregates relevant visual features).
  3. The output $H$ is passed through a residual block and a linear projection to map into the language model's embedding space.
• Result: The adapter compresses a long sequence (e.g., 576 tokens) into a compact set (e.g., 64 tokens) that retains the most semantically relevant information for matching.

C. Compact Joint Encoder

The online inference stage uses a small, efficient language model (e.g., MiniLM) rather than a massive LLM.

• The compressed visual tokens and the text query tokens are concatenated.
• The small language model processes them via self-attention to produce a re-ranking score.
• Benefits: This architecture is modular (any ViT + any small LM), fast, and data-efficient (only the adapter and LM need training).

D. Training Strategy

To ensure strong discriminative performance, EDJE is trained with three objectives:

1. Image-Text Matching (ITM): Binary classification using in-batch hard-negative mining (selecting confusing negatives based on an embedding model).
2. Masked Language Modeling (MLM): Predicting masked text tokens given visual tokens and unmasked text to align features.
3. Text-Embedding Recovery: A projection layer encourages the [CLS] token to recover the embedding of the text encoder, strengthening the global representation.

• Distillation: The compressed variant is distilled from a "local" (uncompressed) teacher model to transfer discriminative capacity.

3. Key Contributions

1. Novel Architecture: Introduction of EDJE, the first practical joint encoder for large-scale retrieval that decouples heavy vision extraction from online inference via offline precomputation and token compression.
2. Token Compression Adapter: A learnable attention mechanism that reduces visual token sequences by ~90% (e.g., 576 $\to$ 64) while preserving semantic alignment, drastically reducing storage and compute.
3. Efficiency-Performance Trade-off: Demonstrates that EDJE can process 50,000 image-text pairs per second with only 49kB of storage per image (for 64 tokens), matching or exceeding the performance of prior joint encoders.
4. Comprehensive Analysis: Extensive ablation studies on token counts, pool sizes, training objectives, and quantization, proving the robustness of the approach.

4. Experimental Results

The authors evaluated EDJE on Flickr30k (zero-shot) and COCO (fine-tuned) benchmarks.

• Performance vs. Embedding Models: EDJE significantly boosts the performance of standard embedding-based pipelines. For example, when paired with CLIP, it improved Recall@1 by up to 15% for image retrieval.
• Performance vs. Prior Joint Encoders:
  • EDJE matches or surpasses state-of-the-art joint encoders (BLIP, BLIP-2, ALBEF) on standard benchmarks.
  • Efficiency: EDJE is up to 53x faster in inference time than prior joint encoders.
  • Storage: EDJE requires ~49kB per image (compressed 64 tokens) compared to ~1.7MB for uncompressed features or full models.
• Scalability: In "Full Dataset" retrieval scenarios (re-ranking against all images in the dataset, not just top-k), EDJE substantially outperforms baselines like LightningDOT.
• Quantization: The compressed tokens are robust to aggressive quantization (down to FP4), further reducing storage with negligible performance loss.

5. Significance and Impact

• Bridging the Gap: EDJE successfully bridges the gap between the high accuracy of joint encoders and the high throughput required for industrial-scale retrieval systems.
• Practical Deployment: By moving vision extraction offline and compressing tokens, EDJE makes it feasible to deploy discriminative re-rankers in real-world applications (e.g., web search, content moderation) where latency and storage costs are critical constraints.
• Paradigm Shift: The paper suggests a new paradigm for Vision-Language Models (VLMs) in retrieval: separating the heavy visual encoding (offline) from the interaction (online), allowing for the use of much larger and more expressive vision backbones without sacrificing speed.
• Future Directions: The authors note that this approach could extend to video retrieval (temporal compression) and multilingual tasks, opening new avenues for efficient multimodal AI.

In summary, EDJE proves that joint modeling does not have to be slow. By intelligently compressing visual features and shifting computation offline, it delivers state-of-the-art retrieval accuracy with the efficiency of embedding-based models.