💬 NLP

Efficient and High-Fidelity Omni Modality Retrieval

The paper introduces OmniRet, the first retrieval model capable of handling complex queries across text, vision, and audio modalities, which achieves efficient and high-fidelity performance through novel attention-based resampling and Attention Sliced Wasserstein Pooling mechanisms, while also establishing a new Audio-Centric Multimodal Benchmark to evaluate omni-modal capabilities.

Original authors: Chuong Huynh, Manh Luong, Abhinav Shrivastava

Published 2026-03-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chuong Huynh, Manh Luong, Abhinav Shrivastava

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific song in a massive library. In the old days, you could only ask the librarian for help using words (e.g., "I want a sad song about rain"). Later, you could show a picture of a rainy street, and the librarian would find songs that matched the mood.

But what if you wanted to say: "I want a song that sounds like this recording of a dog barking, but with the barking replaced by a siren, and the background music should be jazz"?

Current AI systems are like librarians who are great at reading text or looking at pictures, but they get confused when you mix them all together. They struggle to understand complex requests involving Text, Images, Video, and Audio all at once.

This paper introduces OmniRet, a new "Super Librarian" designed to handle these complex, mixed-media requests efficiently and accurately.

Here is how it works, explained through simple analogies:

1. The Problem: The "Too Much Information" Bottleneck

Imagine you ask a librarian to find a book.

The Old Way: To understand a video or a sound clip, the AI breaks it down into thousands of tiny pieces (tokens). It tries to shove all 500+ pieces of a single image into the librarian's brain at once. This is like trying to drink a firehose; it's slow, messy, and the librarian gets overwhelmed.
The Fidelity Problem: To fix the speed issue, other systems just take the "average" of all those pieces. It's like summarizing a 3-hour movie into one sentence: "A guy fights a dragon." You lose all the cool details (the dragon's color, the music, the specific fight moves).

2. The Solution: OmniRet's Two Superpowers

The authors built OmniRet with two special tools to solve these problems.

Superpower A: The "Smart Squeeze" (Shared Media Resampler)

Instead of shoving the whole firehose into the librarian's brain, OmniRet uses a Smart Squeeze.

How it works: It takes that massive pile of 500+ tiny pieces from an image or sound and intelligently compresses them down into a small, fixed set of "key highlights" (like 64 summary points).
The Analogy: Imagine you have a 100-page report. Instead of reading every word, a smart assistant highlights the 10 most important sentences and hands you just those.
Why it's special: This "Squeeze" is shared. It uses the same brainpower for images, sounds, and videos, but it has a little "special note" for each type so it knows how to handle a dog bark differently than a violin note. This makes the system fast and efficient.

Superpower B: The "High-Definition Snapshot" (Attention Sliced Wasserstein Pooling)

Now, how do we make sure we don't lose the fine details when we compress the data?

The Old Way: Taking an average is like looking at a blurry photo.
OmniRet's Way: They use a technique called Attention Sliced Wasserstein Pooling.
The Analogy: Imagine you have a bag of mixed marbles (red, blue, green) and you want to describe the bag to a friend.
- Average Pooling says: "It's a purple bag." (You lost the colors).
- OmniRet says: "I'm going to slice this bag into thin layers. In the first slice, I see mostly red. In the second, mostly blue. In the third, a mix." It creates a 3D map of the marbles rather than just a single color.
The Result: It preserves the "fine-grained" details (like the specific pitch of a siren or the texture of a fabric) while still keeping the data small enough to be fast.

3. The New Training Ground (The ACM Benchmark)

To prove this works, the authors realized there was no good "test" for audio. Existing tests were mostly about pictures and words.

The Gap: No one had a standardized way to test if an AI could understand: "Find a video of a dog, but change the bark to a howl."
The Fix: They created the ACM Benchmark (Audio-Centric Multimodal). They used powerful AI to generate thousands of these complex "change this sound" puzzles.
The Result: When they tested OmniRet, it crushed the competition. It was the only model that could handle the complex "mix-and-match" audio tasks effectively, while still being just as good at standard image and text searches.

Summary: Why This Matters

Think of OmniRet as the universal remote control for the internet.

Before: You needed a different remote for your TV (Video), your Spotify (Audio), and your Google Search (Text). They didn't talk to each other well.
Now: OmniRet is one remote that understands you can say, "Show me a video of a beach, but make the sound like a thunderstorm, and find me a song that matches that mood."

It achieves this by being efficient (not getting overwhelmed by data) and high-fidelity (remembering the tiny details). This brings us one big step closer to AI that can truly understand the messy, multi-sensory world we live in, rather than just reading text or looking at static pictures.

1. Problem Definition

The paper addresses the limitations of current multimodal retrieval systems, which are predominantly restricted to bi-modal interactions (text-vision). While models like CLIP and BLIP excel at text-image tasks, they struggle with universal retrieval involving three or more modalities (text, vision, and audio) simultaneously.

Two critical bottlenecks prevent the development of universal retrieval systems:

Computational Efficiency: Feeding massive token sequences from media encoders (e.g., >500 tokens for a single image or video) directly into Large Language Models (LLMs) creates a computational explosion. This limits batch sizes, which in turn degrades the effectiveness of contrastive learning.
Representation Fidelity: Compressing rich, multi-modal data into a single embedding vector often results in an "information bottleneck," discarding fine-grained details. Standard pooling methods (e.g., mean pooling or using the [EOS] token) blur token-level information, while late-interaction methods (e.g., ColBERT) are too computationally expensive for large-scale retrieval.

2. Methodology: OmniRet Architecture

OmniRet is a unified framework designed to handle composed queries and targets across text, vision, and audio. It employs a frozen LLM as a "universal composer" and introduces two novel modules to solve the efficiency and fidelity challenges.

A. Shared Media Resampler (Efficiency)

To address the token explosion problem, the authors introduce an Attention-based Resampler based on the Perceiver architecture.

Function: It condenses long sequences of media tokens (from visual or audio encoders) into a compact, fixed-size set of latent vectors (e.g., $N=64$ ) before they enter the LLM.
Design: While the core resampler is shared across modalities to enhance generalization, it utilizes modality-specific latents added to the shared queries. This allows the model to adapt to specific modality characteristics while maintaining a unified architecture.
Diversity Loss ( $L_{div}$ ): To ensure the resampled tokens do not collapse into a single representation, a diversity regularization loss is applied. It encourages orthogonality among the output vectors using a sparse sampling mechanism (Dropout) and a smooth $L_1$ loss, ensuring maximal information retention.

B. Attention Sliced Wasserstein Pooling (ASWP) (High-Fidelity)

To overcome the information loss of standard pooling, the authors propose ASWP for aggregating the LLM's output hidden states into a final embedding.

Mechanism: Instead of averaging, ASWP treats the set of LLM tokens as a distribution. It compares this distribution against a set of learnable reference points across multiple 1D projections (slices).
Process:
1. Slicing: Projects the token distribution and reference points onto $L$ random directions.
2. Monge Coupling: Computes the 1D optimal transport distance (Monge coupling) for each slice, creating a "histogram-like" descriptor that preserves fine-grained structural information.
3. Hard Selection: Uses a Straight-Through Maximum (STM) estimator to select the most relevant features from the sliced descriptors, allowing gradients to flow through the discrete selection process.
Benefit: This method retains token-level granularity (unlike mean pooling) while maintaining the efficiency of a single-vector retrieval system (unlike late-interaction models).

C. Training Strategy

Architecture: Uses GTE-Qwen2-1.5B-Instruct as the LLM backbone. Visual inputs use SigLIP encoders; audio uses QwenAudio encoders. Only the projectors, resampler, ASWP, and LoRA adapters are trained; encoders and the LLM are frozen.
Data: Trained on an aggregation of ~6.2 million query-target pairs from 30 diverse datasets (including text, image, video, and audio).
Loss Functions: A combination of Hard-negative InfoNCE loss, Triplet loss, and the Diversity loss.

3. Key Contributions

OmniRet Framework: The first retrieval model capable of handling composed queries spanning text, vision, and audio simultaneously.
Novel Architectural Components:
- Shared Media Resampler: Solves the efficiency bottleneck by compressing media tokens without losing modality-specific sensitivity.
- ASWP: A new pooling technique that bridges the gap between single-vector efficiency and token-level fidelity.
ACM Benchmark (Audio-Centric Multimodal):
- A new benchmark curated from the VGG-Sound dataset to evaluate universal retrieval beyond text-vision.
- Introduces two previously missing tasks: Composed Audio Retrieval (Audio + Text $\to$ Audio) and Audio-Visual Retrieval (Audio $\to$ Image/Video).
- Generated high-quality synthetic modification texts using Gemini 2.5 and validated via human evaluation.
Large-Scale Training Strategy: A two-stage curriculum (warm-up on uni-modal tasks, followed by fine-tuning on all 30 datasets) enabling robust generalization.

4. Experimental Results

The model was evaluated on 13 retrieval tasks (extended M-BEIR), a subset of MMEBv2, and the new ACM benchmark.

M-BEIR Performance: OmniRet achieves state-of-the-art (SOTA) performance on composed retrieval tasks involving images, videos, and audio. It significantly outperforms specialized models (like CLAP for audio or ImageBind for audio-visual) and multi-task baselines (like VLM2VecV2), particularly in composed query scenarios.
MMEBv2 Generalization: Despite not being fully fine-tuned on specific video datasets, OmniRet achieves SOTA results on video retrieval and moment retrieval tasks, demonstrating strong zero-shot generalization.
ACM Benchmark:
- Composed Audio (A,T $\to$ A): OmniRet (23.0 Recall@5) significantly outperforms the strongest baseline (a multi-stage pipeline using Gemma + QwenOmni at 44.6, though the pipeline is computationally heavy and less efficient) and specialized models like CLAP (16.1). Note: The text-based pipeline score in Table 4 seems higher but the paper argues it is inefficient and fails on other tasks.
- Audio-Visual (A $\to$ V/I): OmniRet matches or exceeds the performance of ImageBind and OmniBind, proving it can align audio with visual modalities effectively.
Ablation Studies:
- Removing the Resampler drops performance by ~3.5%.
- Replacing ASWP with Average Pooling causes a massive drop (~29.5%), validating the need for high-fidelity pooling.
- The Diversity Loss is critical; removing it drops performance by 3.1%.

5. Significance

This work represents a significant leap toward Universal Information Retrieval.

Practicality: By solving the efficiency bottleneck, OmniRet makes it feasible to deploy universal retrieval systems in real-world applications (e.g., RAG systems, recommendation engines) that require understanding complex, multi-modal user intent.
Audio Advancement: It fills a critical gap in the field by providing a robust framework and benchmark for audio-centric retrieval, a modality historically underserved compared to text and vision.
Architectural Insight: The introduction of ASWP provides a new paradigm for embedding aggregation, demonstrating that high-fidelity representations can be achieved without the prohibitive costs of late-interaction models.

In conclusion, OmniRet successfully unifies text, vision, and audio into a single, efficient, and high-fidelity embedding space, setting a new standard for omni-modal retrieval systems.