Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio
The paper introduces the Fusion Embedding family, a unified model that integrates audio into a frozen vision-language embedding space using lightweight, modality-specific adapters to enable zero-shot cross-modal retrieval across text, image, video, and audio without updating the base parameters.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are building the ultimate library for the entire internet. Right now, if you want to find a song, you have to ask a music librarian; if you want a photo, you ask a photo librarian; and if you want a video, you ask a video librarian. They all speak different languages and keep their books in separate rooms. This makes searching for something that mixes these things—like "a video of a dog barking at a full moon"—incredibly hard.
To solve this, scientists use something called "embeddings." Think of an embedding as a unique ID card or a set of GPS coordinates for a piece of data. In a perfect world, if you have a picture of a dog and a sentence that says "a dog," they should get the exact same GPS coordinates, even though one is a picture and the other is text. This lets a computer understand that they are the same thing. The big challenge right now is that while we have great GPS systems for text, images, and video, we are terrible at mapping sound. Most systems that understand sound are like specialists who only speak "noise" and can't talk to the text or image librarians at all.
This paper introduces a new family of tools called Fusion Embedding that finally puts text, images, video, and audio into the same single map. The researchers didn't try to rebuild the whole library from scratch. Instead, they took a very powerful, pre-built library that already knew how to handle text, images, and video, and they figured out a clever way to add a "sound department" without breaking anything that was already working.
The Magic of the "Frozen" Library
The secret sauce here is that the researchers kept the main library "frozen." Imagine the main library is a giant, perfect encyclopedia written by a master scribe. Usually, if you want to add a new section (like sound), you have to rewrite parts of the encyclopedia, which risks changing the meaning of the old words.
The authors decided: "No rewriting allowed." They kept the encyclopedia exactly as it was, down to the last letter. Instead, they built a small, special translator booth right at the entrance.
Generation 1 is this first translator. It's a tiny, 16.4-million-parameter connector (think of it as a small, efficient robot) that sits between the audio tower (the part that hears sound) and the frozen encyclopedia. When you feed it a sound, the robot translates the noise into a format the encyclopedia understands. It doesn't change the encyclopedia; it just speaks its language. Surprisingly, this simple setup worked incredibly well. It could match a sound to a text description, and even better, it could match a sound to an image it had never seen before, all without ever being shown a single picture of a sound. It's like teaching a person who only speaks French to understand a new language just by showing them a dictionary, without ever teaching them the grammar rules of the new language.
Generation 2 asked, "Can we do even better without breaking the 'no rewriting' rule?" They realized that while the translator booth was good, the encyclopedia itself had some unused brainpower that could help with sound. So, they added "modality-gated deep adapters."
Think of these as secret, invisible doors inside the encyclopedia's pages. These doors only open when a sound is being processed. When a text or image comes in, the doors stay shut, and the book is read exactly as it always was. When a sound comes in, the doors open, and a special 44.2-million-parameter helper module jumps in to help the book understand the sound better. The result? The book's output for text and images remains bit-for-bit identical to the original frozen version. If you ran the same text through the system before and after adding the sound feature, the computer would see no difference at all. This guarantees that every single search result you've ever saved for text, images, or video remains perfectly valid.
What They Found (and What They Didn't)
The team tested this system on massive datasets of sounds and captions. They found that by sticking to their "frozen base" rule, they could achieve state-of-the-art results for audio-to-text matching, reaching a score of 0.741 on a standard test called AudioCaps. This is a huge deal because most other systems that do this well have to retrain their entire brain, which means they often forget how to do other things or break their old results.
However, the paper also tells us what doesn't work. The researchers tried a few things that seemed like they should help but actually made things worse:
- Rewriting the captions: They tried using a super-smart AI to rewrite the training descriptions to make them "cleaner." Surprisingly, this made the system perform worse. It turns out, the system learned better from the original, messy human descriptions than from polished AI ones.
- Using a "stronger" sound tower: They tried swapping their sound encoder for one that is usually considered the best in the world. But when plugged into their frozen system, this "stronger" tower actually performed worse than their chosen one. This suggests that what makes a good sound encoder for a standalone system isn't necessarily what makes a good sound encoder for a frozen-library system.
- Making the connector too wide: They tried making the translator robot bigger and wider. Instead of getting smarter, it just started memorizing the training data and failed to generalize.
The "Protocol" Surprise
One of the most interesting discoveries wasn't about the model itself, but about how they talked to it. The researchers found that the way you feed text into the system matters more than you'd think. If you feed the system text in a "bare" format (just the words), it performs poorly. But if you feed it the text in the exact "chat template" format the original encyclopedia was trained on (with specific instructions and formatting), the performance jumps by a massive 14.5 points.
It's like trying to ask a librarian a question. If you just shout "Dog!" they might not understand. But if you say, "Hello librarian, I am looking for a picture of a dog," in the exact polite format they were taught, they instantly find it. The paper shows that if you don't respect the original training format, you can lose double-digit points in performance, effectively making your system much dumber.
The Bottom Line
The Fusion Embedding family proves that you don't need to rebuild the whole house to add a new room. By using a frozen base and adding smart, gated connectors, they created a unified space where text, images, video, and audio all live together.
- Generation 1 showed that a small connector is enough to make audio a "first-class citizen" in the library.
- Generation 2 showed that you can add even more brainpower for audio without ever changing a single bit of the text or image results.
The result is a system that can take a sound and find the matching video, or take a video and find the matching sound, all while keeping the original text and image search results perfectly intact. It's a step toward a future where you can ask your computer, "Show me videos of rain," and it understands that "rain" is a sound, a visual, and a word, all at once, without needing a different app for each.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.