WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLM

This paper introduces WAVE, the first LLM-based embedding model that unifies text, audio, and video into a single representation space through hierarchical feature fusion and joint multi-task training, achieving state-of-the-art performance in cross-modal retrieval and prompt-aware multimodal question answering.

The Gist

Imagine you have a massive library where books, movies, music, and podcasts are all stored in completely different rooms. Currently, if you want to find a movie that sounds like a specific song, or a book that matches the mood of a video, you have to hire a different librarian for each room. They speak different languages and don't talk to each other.

WAVE is like hiring a super-intelligent, multilingual "Universal Librarian" who can walk into any room, understand the content instantly, and translate it all into a single, universal language that everyone understands.

Here is a breakdown of what this paper is about, using simple analogies:

1. The Problem: The "Tower of Babel" of Media

Right now, computers are good at understanding text, and they are getting better at understanding images and video. But audio (sound) and video (moving pictures) are often treated as separate worlds.

• The Old Way: You have one computer brain for text, another for pictures, and a third for sound. They don't really know how to talk to each other. If you ask, "Find me a video that sounds like jazz," the text brain and the sound brain might get confused.
• The WAVE Solution: WAVE is a new type of AI (based on a powerful model called Qwen2.5-Omni) that learns to put text, audio, silent video, and full video all into one shared "mental space." It's like having a single dictionary where the word "sunset" has the same meaning whether you read it, see a video of it, or hear the wind blowing during it.

2. How It Works: The "Swiss Army Knife" Architecture

The paper describes a few clever tricks WAVE uses to become this Universal Librarian:

• The Dual-Ear System: Humans have two ears to hear different things (like a voice vs. background noise). WAVE uses a dual-encoder for audio. One part listens for "speech" (like a person talking), and the other listens for "events" (like a dog barking or rain falling). This lets it understand the full richness of sound.
• The "Layer Cake" Fusion: Usually, AI models look at the very last layer of their brain to make a decision. WAVE is smarter; it looks at every layer of its brain, from the bottom (simple details) to the top (complex ideas). It then mixes all these layers together like a chef blending ingredients in a smoothie. This ensures it doesn't miss the small details or the big picture.
• The "Prompt" Magic: Most AI models just give you a generic summary. WAVE is prompt-aware. Think of it like a personal assistant.
  • Generic AI: "Here is a summary of the video."
  • WAVE: "Here is a summary of the video specifically focusing on the colors" or "specifically focusing on the sad music."
  • Because it can follow instructions, it creates a "customized" version of the video's meaning depending on what you ask.

3. The Training: Learning by Doing Everything at Once

To teach WAVE, the researchers didn't just show it one type of data. They threw everything at it at the same time:

• They showed it videos and asked it to find matching text.
• They showed it audio and asked it to find matching videos.
• They asked it questions about what it saw and heard.

The Analogy: Imagine training a student by making them study math, history, and music simultaneously in the same class. Instead of being a specialist who only knows math, this student learns how math concepts relate to music rhythms and historical patterns. The paper found that this "joint training" made WAVE much smarter than models trained on just one subject.

4. The Results: Why It Matters

The paper tested WAVE on some very difficult challenges:

• The "Any-to-Any" Game: Can you find a video using a sound clip? Can you find a song using a text description? WAVE is currently the best at this. It's like being able to find a movie just by humming a tune from it.
• The "Quiz Show": When asked specific questions about a video (e.g., "What color was the car in the background?"), WAVE got much better scores than other models because it could focus its "attention" based on the question.
• Beating the Giants: It beat existing top-tier models (even some from big tech companies) on video and audio benchmarks.

The Big Takeaway

WAVE is a breakthrough because it stops treating sound, sight, and words as separate things. It creates a unified language for all senses.

In everyday terms:
Before WAVE, if you wanted to find a video of a "happy dog playing in the rain," you had to search for "dog," then "rain," then hope the algorithm guessed you wanted them together.
With WAVE, you can just say, "Show me the happy dog in the rain," and it understands the feeling and the context of all those elements combined, instantly finding exactly what you need. It's the first step toward AI that truly understands the world the way humans do—through a mix of sight, sound, and language all at once.

Technical Summary

1. Problem Statement

Current multimodal embedding models typically rely on separate encoders for each modality (text, image, audio, video) aligned in a shared space. While effective, this approach often struggles with:

• Dynamic Modalities: Limited exploration of audio and synchronized audio-visual streams compared to static images.
• Semantic Alignment: Difficulty in achieving true cross-modal interoperability and holistic representation when modalities are processed separately.
• Prompt Awareness: Most embedding models generate task-agnostic representations, failing to adapt to specific user instructions or questions (e.g., in Multimodal Question Answering).
• Unified Space: The lack of a single model capable of producing unified embeddings for text, silent video, audio, and synchronized audio-visual inputs simultaneously.

2. Methodology

WAVE is a unified, versatile audio-visual embedding model built upon the Qwen2.5-Omni (7B parameter) foundation. It employs a novel architecture and training strategy to create a single semantic space for all modalities.

A. Model Architecture

• Input Processing: WAVE accepts four input configurations: Text-only, Visual-only, Audio-only, and Audio-Visual.
• Encoders:
  • Visual: Uses a pre-trained visual encoder to extract frame features.
  • Audio: Employs a dual-encoder strategy. A speech encoder captures speech-related tokens, while a separate audio encoder (BEATs) captures environmental sound events. These are interleaved to form a unified auditory sequence.
  • Text: Tokenized using the LLM's native embedding layer.
• Token Interleaving & Positioning:
  • For audio-only, speech and event tokens are interleaved 1:1.
  • For audio-visual, visual and auditory tokens are segmented by time and interleaved.
  • TMRoPE: Utilizes Time-Aligned Multimodal Rotary Position Embedding (from Qwen2.5-Omni) to ensure precise temporal alignment across modalities.
• Feature Fusion (Hierarchical):
  • Instead of standard last-token pooling from only the final layer, WAVE aggregates the last output tokens from all LLM layers.
  • These multi-layer features are concatenated and passed through a lightweight 2-layer MLP (Fusion Module) to produce the final unified embedding. This captures both low-level perceptual cues and high-level semantic abstractions.
• Prompt Conditioning: Non-text inputs are always accompanied by a text prompt (instruction) to guide the LLM, enabling prompt-aware embeddings.

B. Training Strategy

WAVE utilizes a joint multi-modal, multi-task training approach with two primary objectives:

1. Multimodal Retrieval (Contrastive Learning):
   • Trains on "any-to-any" cross-modal pairs (e.g., text-to-video, video-to-audio).
   • Uses symmetric InfoNCE loss with in-batch negative sampling to align source and target embeddings.
2. Question Answering (QA) (Discriminative Learning):
   • Trains the model to generate embeddings conditioned on specific questions.
   • Uses a cross-entropy loss to maximize the similarity between the query embedding and the correct answer while minimizing similarity with distractor answers.

Data: The model is trained on 4.9M samples across diverse datasets (Panda-70M, AudioSet, WavCaps, etc.), covering video-text, video-audio, and audio-text retrieval, as well as video QA.

3. Key Contributions

1. First Unified Audio-Visual Embedding MLLM: WAVE is the first model to produce unified embeddings for text, audio, silent video, and synchronized audio-visual inputs within a single semantic space.
2. Prompt-Aware Embeddings: By leveraging the instruction-following capabilities of the underlying MLLM, WAVE generates embeddings that adapt to user instructions. This significantly improves performance in downstream tasks like Multimodal QA, where static embeddings fail.
3. Hierarchical Feature Fusion: The proposal to aggregate and fuse features from all LLM layers (rather than just the last layer) via an MLP yields more robust representations, capturing both fine-grained details and abstract semantics.
4. Dual Audio Encoder: The separation of speech and environmental sound processing enhances the expressiveness of audio embeddings.
5. New Benchmark: Introduction of a benchmark for versatile audio-visual learning to evaluate cross-modal capabilities.

4. Experimental Results

WAVE achieves State-of-the-Art (SOTA) performance across multiple benchmarks, outperforming existing open-source models and even industrial-grade baselines.

• Video Benchmarks (MMEB-v2):
  • Achieved an Overall score of 59.9, surpassing the industrial model Seed-1.6-Embedding (55.3) and other open-source models like LamRA (35.0) and GME (38.4).
  • Led in Classification (CLS), QA, and Retrieval (RET) sub-tasks.
• Audio & Audio-Visual Retrieval:
  • Outperformed separate-encoder retrieval models on AudioCaps (44.2 vs 42.2) and Clotho (25.6 vs 21.5).
  • Demonstrated superior capability in Video-to-Audio and Video-to-Music retrieval, tasks where text is bypassed, proving the strength of the unified space.
• Multimodal QA:
  • On MMEB-v2 Video QA, WAVE achieved 72.5% accuracy when using separate questions as prompts, significantly outperforming Seed-1.6 (60.9%).
  • Crucially, when using a generic prompt ("Describe the video"), performance dropped drastically, highlighting the model's unique ability to generate task-specific, prompt-aware embeddings.
• Ablation Studies:
  • Joint Training: Jointly trained models outperformed specialist models trained on single modality pairs in 7 out of 8 tasks, confirming positive cross-modal knowledge transfer.
  • Feature Fusion: The "All-layer last token MLP fusion" strategy consistently outperformed standard last-layer pooling and weighted sum approaches.

5. Significance

• Paradigm Shift: WAVE moves the field from separate encoders aligned in a shared space to a single, unified MLLM that natively processes and embeds all modalities.
• Versatility: It bridges the gap between retrieval and reasoning, enabling embeddings that are not just static vectors but dynamic representations conditioned on user intent.
• Foundation for Applications: By establishing a robust baseline for "any-to-any" cross-modal interaction, WAVE opens new possibilities for complex applications such as cross-modal search, content recommendation, and advanced multimodal reasoning systems.
• Open Science: The authors release code, checkpoints, and a new benchmark, fostering reproducibility and further research in unified audio-visual learning.