AudioX: A Unified Framework for Anything-to-Audio Generation

This paper introduces AudioX, a unified framework for anything-to-audio generation that leverages a Multimodal Adaptive Fusion module and a large-scale 7-million-sample dataset (IF-caps) to achieve superior performance in generating audio from diverse multimodal inputs like text, video, and audio signals.

The Gist

Imagine you have a magical radio that can create any sound you can imagine. But here's the catch: most radios today are very picky. One radio only plays sounds if you describe them with words (Text-to-Audio). Another only works if you show it a video (Video-to-Audio). A third only plays music if you give it a specific genre tag. They are like specialized chefs: one only makes pizza, another only makes sushi, and they can't swap recipes.

AudioX is the "Master Chef" of sound. It's a new AI framework that can take any combination of clues—words, videos, or even snippets of other audio—and cook up the perfect sound or music track.

Here is a simple breakdown of how it works, using some everyday analogies:

1. The Problem: The "Specialist" Bottleneck

Before AudioX, if you wanted to make a sound effect for a movie scene where a dog barks at a car, you might need one AI to understand the video of the car, another to understand the text "dog barking," and a third to actually make the sound. It was clunky, like trying to build a house by hiring a plumber for the roof and an electrician for the foundation. They didn't talk to each other well.

2. The Solution: The "Universal Translator" (AudioX)

AudioX is built to be a unified framework. Think of it as a super-smart conductor in an orchestra.

• The Inputs: You can whisper a script (Text), show a video clip (Video), or play a partial melody (Audio).
• The Magic: AudioX listens to all these different inputs at once and says, "Okay, I understand the story, the visual, and the mood. Let's create the sound."

3. The Secret Sauce: The "Smart Mixer" (MAF Module)

The paper introduces a special component called the Multimodal Adaptive Fusion (MAF) module.

• The Analogy: Imagine you are at a noisy party with three friends talking to you at once. One is shouting, one is whispering, and one is singing. If you try to listen to all of them equally, you get confused.
• How AudioX works: The MAF module is like a smart sound engineer at the party. It has a "gate" that turns down the volume on the shouting friend (noise) and turns up the volume on the whispering friend (important details). It figures out which clues are most important for the specific sound you want and blends them perfectly so they don't clash. This ensures the AI doesn't get confused when you give it both a video and a text prompt.

4. The Training Data: The "Giant Library" (IF-caps)

To teach this "Master Chef" to be so good, the researchers had to feed it a massive amount of data. Existing libraries were like having a cookbook with only pizza recipes.

• The Innovation: They built a new library called IF-caps (Instruction-Following captions). It contains 7 million samples!
• The Process: They didn't just copy-paste old data. They used a "two-step cooking process":
  1. The Head Chef (Gemini AI): Looked at a video and wrote a very detailed, high-quality description of the sounds (e.g., "A dog barks twice, then a car drives by").
  2. The Sous Chef (Qwen AI): Took that description and rewrote it in 100 different ways to teach the model that "a dog barking" and "a canine making a noise" mean the same thing.
• Result: This taught the AI to understand not just what sound to make, but how many, when, and in what order.

5. The Result: Following Instructions Like a Pro

The most impressive part of AudioX is its instruction-following.

• Old AI: If you asked for "a dog barking," it might make a dog bark, but maybe 5 times, or maybe it barks before the car arrives. It's like a student who hears the assignment but misses the details.
• AudioX: If you say, "A dog barks twice, then a car drives by after 3 seconds," AudioX does exactly that. It's like a student who reads the instructions, highlights the key numbers, and follows them perfectly.

Why This Matters

In the real world, this means:

• Game Developers can type a description of a scene and get the perfect background music and sound effects instantly.
• Filmmakers can show a silent video clip and get realistic sound effects that match the action perfectly.
• Musicians can hum a tune or describe a mood ("sad, slow, with a cello") and get a full song generated.

In short: AudioX is the first AI that doesn't just "guess" sounds based on one clue. It listens to the whole story, understands the details, and creates high-quality audio that follows your instructions like a pro. It's the difference between a random noise machine and a professional sound designer in your pocket.

Technical Summary

1. Problem Statement

The field of audio and music generation has traditionally relied on specialized models constrained by specific input modalities (e.g., text-to-audio, video-to-audio) and output domains (sound effects vs. music). These siloed approaches face two primary limitations:

• Lack of Unification: Existing models struggle to handle flexible combinations of multimodal inputs (text, video, audio) within a single framework, limiting their adaptability to complex, real-world scenarios.
• Data Scarcity: High-quality, large-scale datasets suitable for training unified, multimodal systems are scarce. Most existing datasets are task-specific (e.g., only text-to-audio) and lack the fine-grained, structured annotations required for precise instruction-following and cross-modal alignment.
• Weak Instruction Following: Current models often fail to adhere to complex, fine-grained instructions regarding sound event counts, temporal ordering, and duration, despite achieving high audio fidelity.

2. Methodology

The authors propose AudioX, a unified framework designed for "anything-to-audio" generation. The methodology consists of three core components:

A. Model Architecture: Diffusion Transformer with Multimodal Adaptive Fusion (MAF)

• Backbone: AudioX utilizes a Diffusion Transformer (DiT) backbone, which has demonstrated superior performance in high-fidelity audio synthesis compared to autoregressive models.
• Input Processing: The model accepts diverse inputs: Video ($X_v$), Text ($X_t$), and Audio ($X_a$). Each modality is processed by specialized encoders (CLIP-ViT for video, T5 for text, and a custom Audio Autoencoder).
• Multimodal Adaptive Fusion (MAF): This is the core innovation. To prevent interference between modalities and ensure effective alignment, the MAF module employs a lightweight mechanism:
  1. Gating: Initial feature embeddings pass through gates to filter noise and reweight informative cues.
  2. Cross-Attention: Learnable queries (organized as modality-specific experts) attend to the gated embeddings to aggregate evidence across streams.
  3. Self-Attention & Residual Update: A self-attention layer consolidates the context, and the refined information is dispatched back to modality paths via residual updates.
  4. Fusion: The output is a unified condition embedding ($H_c$) that guides the DiT generation process.

B. Dataset Construction: IF-caps

To address data scarcity, the authors constructed IF-caps (Instruction-Following captions), a large-scale dataset containing over 7 million samples (1.3M general audio, 5.7M music).

• Pipeline: A two-stage annotation pipeline was used:
  1. High-Quality Initialization: Gemini 2.5 Pro generates holistic captions and structured fields (sound event classification, counts, temporal relations) for video-audio clips.
  2. Scalable Augmentation: Qwen2-Audio is used to augment these annotations with diverse linguistic variations based on the structured fields, increasing data diversity while managing costs.
• Annotation Schema: The dataset includes fine-grained fields such as category, count, Sound Event Detection (SED) timestamps, and time_relation (ordering), enabling the model to learn precise temporal and compositional control.

C. Training Strategy

• Unified Training: The model is trained on a mixture of tasks (Text-to-Audio, Video-to-Audio, Text+Video-to-Audio, Audio Inpainting, Music Completion) using zero-padding for missing modalities.
• Cross-Modal Regularization: The authors observe that high-quality textual supervision acts as a regularizer, reducing alignment noise and improving performance even on non-text tasks (e.g., Video-to-Audio).

3. Key Contributions

1. Unified Framework (AudioX): A single model capable of generating both sound effects and music from flexible combinations of text, video, and audio inputs, overcoming the constraints of specialist models.
2. MAF Module: A novel, lightweight architectural component that effectively fuses multimodal inputs, enhancing cross-modal alignment and instruction-following capabilities.
3. IF-caps Dataset: A massive, high-quality dataset with fine-grained, structured annotations designed specifically to train unified models with strong instruction-following abilities.
4. T2A-bench Benchmark: The introduction of a new benchmark and automated evaluation pipeline to rigorously assess fine-grained instruction following (category, count, ordering, and timestamp accuracy).

4. Experimental Results

AudioX was benchmarked against state-of-the-art (SOTA) models across multiple tasks and datasets (AudioCaps, VGGSound, MusicCaps, V2M-bench, etc.).

• Overall Performance: AudioX achieves SOTA or highly competitive results across all tasks (T2A, V2A, T2M, V2M, Inpainting, Completion). Notably, it outperforms specialist models on out-of-domain datasets (e.g., AVVP), demonstrating strong generalization.
• Instruction Following: On the new T2A-bench and AudioTime benchmarks, AudioX significantly outperforms all baselines in fine-grained control metrics:
  • Category Accuracy: 34.20% (vs. 32.40% for the next best).
  • Count Accuracy: 12.40% (vs. 9.80%).
  • Ordering Accuracy: 23.60% (vs. 19.80%).
  • Timestamp Accuracy: 28.20% (vs. 21.80%).
• Subjective Evaluation: In a user study with 10 audio experts, AudioX achieved the highest scores for Overall Quality (OVL) and Relevance (REL) across most tasks.
• Ablation Studies:
  • Removing the MAF module leads to significant performance drops, confirming its necessity for multimodal fusion.
  • Training on the full GeminiCap-aug pipeline (vs. raw labels or single-stage generation) yields the best results, validating the data curation strategy.
  • The "cross-modal regularization" effect was empirically verified: improving textual supervision quality directly boosted Video-to-Audio performance.

5. Significance

This work represents a significant leap forward in generative audio by moving from specialized, single-modality models to a generalist, unified framework.

• Paradigm Shift: It demonstrates that a single model can master diverse generation tasks (sound effects, music, inpainting) without task-specific fine-tuning.
• Instruction Following: It establishes a new standard for controllability, proving that audio generation can be precisely directed by complex natural language instructions regarding timing, count, and sequence.
• Data-Centric Insight: The paper highlights that high-quality, structured data is the critical bottleneck for unified multimodal models, offering a blueprint for future dataset construction in the field.
• Practical Application: The ability to generate synchronized, high-fidelity audio from video, text, or partial audio inputs opens new avenues for film production, game development, and accessibility tools.