⚡ electrical engineering

MMAudioSep: Taming Video-to-Audio Generative Model Towards Video/Text-Queried Sound Separation

MMAudioSep is a generative model that leverages a pretrained video-to-audio foundation to efficiently achieve superior video- and text-queried sound separation while retaining its original generation capabilities.

Original authors: Akira Takahashi, Shusuke Takahashi, Yuki Mitsufuji

Published 2026-04-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Akira Takahashi, Shusuke Takahashi, Yuki Mitsufuji

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are at a bustling party. There's music playing, people are laughing, a dog is barking, and someone is clinking glasses. It's a chaotic mix of sounds. Now, imagine you want to hear only the dog barking, or only the music, without the rest of the noise.

This is the problem of Sound Separation. Usually, computers are bad at this unless you give them a very specific, narrow job.

Enter MMAudioSep, a new AI model from Sony that acts like a "super-listener" with a magical pair of headphones. Here is how it works, explained simply:

1. The "Chef" Who Learned to Cook Before Learning to Clean

Most sound-separation AI models are like apprentices who start from scratch. They have to learn what a dog sounds like, what a guitar sounds like, and how to separate them all over again.

MMAudioSep is different. It starts as a Video-to-Audio Generator. Think of this as a "Cinematic Chef." This chef has already spent years watching thousands of videos and learning how to create the perfect sound effects to match the action on screen. If a video shows a dog running, this chef knows exactly what a dog running sounds like.

The researchers asked a clever question: "If this chef knows exactly what a dog sounds like, can we just ask them to 'find' the dog in a messy recording instead of 'creating' it?"

The answer is yes. They took this pre-trained "Cinematic Chef" and gave it a little bit of extra training (fine-tuning) to teach it how to listen to a messy mix and pick out the specific sound it was asked for.

2. The Magic Request: "Show Me the Dog!"

You can ask this model for a specific sound in two ways, just like asking a friend for help:

Text Query: You type, "I want to hear the dog."
Video Query: You show the model a video clip of a dog, and it says, "Ah, I see the dog. I will isolate that sound."

It's like having a smart assistant who can look at a video or read your note, then instantly tune their radio to only that specific station, muting everything else.

3. The "Magic Soup" Analogy

Imagine a bowl of soup where you've thrown in carrots, peas, and noodles all mixed together.

Old AI: Tries to guess which bits are carrots by tasting the whole bowl blindly.
MMAudioSep: It's like a chef who already knows exactly what a carrot tastes like because they've cooked with them a million times. When you hand them the mixed soup and say, "Find the carrots," they don't need to guess. They use their deep knowledge of carrots to instantly identify and pull them out, leaving the peas and noodles behind.

4. The Best Part: It Can Still Cook!

Usually, when you train a model to do a new job (like separating sounds), it forgets its old job (making sounds). It's like a chef who learns to wash dishes so well they forget how to cook.

But MMAudioSep is special. Even after learning to separate sounds, it didn't forget how to generate sounds.

If you give it a video of a drum solo and no audio, it can still create the drum sounds from scratch.
If you give it a video of a drum solo and a messy audio mix, it can separate the drums.

It's a "Swiss Army Knife" AI. It hasn't lost its original superpower; it just added a new one.

Why Does This Matter?

Better Quality: Because it learned from a massive amount of video and audio data first, it understands the "vibe" of sounds better than models that just look at audio waves.
Flexibility: You can ask for sounds using text or video, making it very easy to use.
Efficiency: It didn't need to be built from the ground up. It stood on the shoulders of a giant (the original video-to-audio model) to do something new.

In short: MMAudioSep is a smart AI that learned to create movie soundtracks first, and then used that expert knowledge to become the world's best sound detective, able to pick out any specific noise from a chaotic crowd just by looking at a video or reading a note.

1. Problem Statement

Sound separation (SS) is the task of isolating specific sound sources from a mixed audio signal. While traditional approaches have been discriminative (predicting masks or filters), recent trends involve generative models. However, two distinct research fields have developed independently:

Generative Sound Separation: Models that generate separated audio based on text or image queries (e.g., FlowSep, AudioSep).
Video-to-Audio (V2A) Generation: Models that synthesize audio corresponding to input video and text conditions (e.g., MMAudio).

The core problem addressed is the lack of cross-domain knowledge transfer between these fields. Specifically, the authors hypothesize that the rich multimodal knowledge (visual-textual-audio relationships) learned by a high-quality V2A foundation model can be effectively adapted for sound separation tasks, potentially outperforming models trained from scratch or those limited to single-modality queries.

2. Methodology: MMAudioSep

The authors propose MMAudioSep, a generative model for video/text-queried sound separation built upon the pretrained MMAudio foundation model.

Architecture & Core Mechanism

Base Model: The system utilizes MMAudio, a flow-matching based generative model that synthesizes high-fidelity audio from video and text inputs. It operates in a latent space using a Variational Autoencoder (VAE) and a Multimodal Diffusion Transformer (MM-DiT).
Input Modification (Channel-Concatenation Conditioning):
- Standard V2A generation starts with Gaussian noise ( $x_0$ ).
- For separation, MMAudioSep introduces the mixture audio ( $x_m$ ) as an additional input condition.
- Mechanism: The latent vectors of the mixture mel-spectrogram ( $x_m$ ) and the Gaussian noise ( $x_0$ ) are concatenated along the channel dimension before entering the transformer blocks.
- This allows the model to process the mixture and the target separated signal in parallel within the flow-matching framework. The noise addition process only affects the target latent vector, while the mixture channel remains static.
Training Strategy (Fine-Tuning):
- Instead of training from scratch, the authors fine-tune the pretrained MMAudio model.
- Frozen Parameters: The visual and text encoders (CLIP, Synchformer) and the VAE/Vocoder remain frozen to preserve the original multimodal knowledge.
- Learnable Parameters: The audio projection layer, multimodal transformer blocks, and specific conditioning layers are updated using the Conditional Flow Matching (CFM) objective.
- Dataset: Trained on a combination of VGGSound (Video+Audio+Label), AudioCaps, Clotho, and WavCaps (Audio+Text), totaling ~2,500 hours of data.

3. Key Contributions

Novel Fine-Tuning Approach: The paper proposes the first method to adapt a pretrained V2A generative model for video/text-queried sound separation, demonstrating that foundation models can be repurposed for downstream separation tasks.
Superior Performance: MMAudioSep outperforms existing state-of-the-art separation models (both discriminative and generative) across multiple metrics, particularly when leveraging both video and text queries.
Dual Functionality Retention: A critical finding is that after fine-tuning for separation, the model retains its original V2A generation capabilities. This proves the versatility of foundation sound models for multiple downstream tasks without catastrophic forgetting.
Evaluation Framework: The authors advocate for using generative metrics (FAD, CLAP scores, IB-Score) rather than traditional separation metrics (SDR) for generative separation tasks, providing a more holistic assessment of audio quality and semantic alignment.

4. Experimental Results

The model was evaluated on the VGGSound-Clean and MUSIC datasets, comparing against baselines like AudioSep (discriminative, text-only) and FlowSep (generative, text-only).

Sound Separation Performance:
- Multimodal Advantage: MMAudioSep significantly outperforms baselines when using both video and text queries.
- Text-Only: Even with only text queries, MMAudioSep (fine-tuned) outperforms AudioSep and FlowSep.
- Metrics: It achieved lower Fréchet Audio Distance (FAD) and higher CLAP scores (semantic alignment) and CLAP-A (similarity to ground truth) compared to baselines. For example, on the MUSIC dataset, MMAudioSep achieved a CLAP-A of 91.48% (vs. 86.69% for AudioSep).
- Visual Quality: Spectrogram visualizations show that MMAudioSep preserves fine-grained spectral details and reduces artifacts better than AudioSep.
Video-to-Audio Generation Retention:
- When tested on V2A generation (using white noise as the mixture input), MMAudioSep maintained strong performance.
- While slightly lower than the original MMAudio (due to fine-tuning trade-offs), it outperformed several dedicated V2A baselines (e.g., ReWaS, Seeing&Hearing) in specific metrics like Inception Score (IS) and IB-Score (AV semantic alignment).
- Parameter Efficiency: The "frozen" fine-tuning variant (540M learnable parameters) performed comparably to the "unfrozen" variant (1.03B parameters), suggesting efficient knowledge transfer.

5. Significance and Future Work

Foundation Model Potential: The study validates that generative foundation models trained on massive multimodal datasets possess transferable knowledge that benefits specialized tasks like sound separation, challenging the notion that separation requires purely discriminative training.
Efficiency: Leveraging pretrained weights allows for efficient training without the need for massive datasets from scratch, reducing computational costs.
Future Directions: The authors aim to extend MMAudioSep to universal sound separation (handling broader sound categories) and explore its application in other downstream tasks like sound event detection and visual-acoustic matching.

In summary, MMAudioSep successfully bridges the gap between audio generation and separation, demonstrating that a single generative foundation model can effectively handle both synthesizing audio from video and isolating sounds based on visual/textual queries.