Technical Summary: AudioX-Turbo

Problem Statement

The field of audio and music generation faces three primary challenges that hinder the development of versatile, real-time systems:

1. Lack of Unified Modeling: Existing models are typically specialized for single conditioning modalities (e.g., text-to-audio or video-to-audio) or single output domains (sound effects vs. music), lacking a unified framework to handle flexible multimodal combinations.
2. Data Scarcity: High-quality, large-scale datasets suitable for training unified "anything-to-audio" models are scarce. Existing datasets are often task-specific, providing supervision for only one modality, which limits the training of robust, generalist models.
3. Inference Efficiency: State-of-the-art diffusion-based audio generation models require tens to hundreds of sequential function evaluations (NFE) to solve the underlying ordinary differential equations (ODEs). This high sampling cost results in prohibitive inference latency, making them unsuitable for real-time applications like interactive content creation. Furthermore, aggressive step-distillation techniques often degrade cross-modal alignment and instruction-following capabilities.

Methodology

The authors propose AudioX-Turbo, a unified and efficient framework for generating audio and music from diverse multimodal inputs (text, video, and audio). The framework operates on a teacher–student paradigm involving three core components:

1. Data Construction: IF-caps-Pro

To address data scarcity, the authors constructed IF-caps-Pro, a large-scale dataset comprising approximately 9.2 million samples. The construction follows a two-stage pipeline:

• Stage 1 (Source Curation): Aggregates video-audio pairs from public corpora (VGGSound, AudioSet-Strong) and constructs a new high-quality video-music corpus (V2M-500K) via a multi-step filtering pipeline (removing broken tracks, non-musical content, and static visuals).
• Stage 2 (Multimodal Annotation): Employs a cascade of Large Language Models (Gemini 2.5 Pro and Qwen2-Audio) to generate fine-grained, structured annotations. This includes global captions, sound event classification, counts, ordering relations, and musical attributes (genre, instrumentation, tempo) for 10-second clips.

2. Unified Pretraining: AudioX-Base (Teacher)

The teacher model, AudioX-Base, is built on a Multimodal Diffusion Transformer (MMDiT) architecture trained via Flow Matching.

• Multimodal Adaptive Fusion (MAF): A lightweight module designed to align diverse inputs. It uses gating mechanisms to filter and reweight features from each modality, followed by cross-attention with learnable queries to aggregate evidence across streams. This reduces cross-modal interference and enhances instruction-following.
• Training Objective: The model minimizes the mean squared error between the predicted velocity field and the target vector field in a continuous-time flow matching formulation, unifying multimodal inputs into a latent space.

3. Efficient Distillation: AudioX-Turbo (Student)

To achieve real-time inference, the teacher is distilled into a few-step student model using Distribution Matching Distillation (DMD) adapted for flow matching.

• Distillation Process: The student is trained to match the output distribution of the teacher using a "denoise-renoise" rollout paradigm. An auxiliary fake model tracks the student's evolving distribution to provide distribution-matching signals.
• Diffusion-Based Discriminator: To preserve high-frequency acoustic textures and perceptual realism often lost in few-step sampling, a discriminator is employed. It reuses the frozen early transformer blocks of the teacher as a feature backbone, operating on slightly perturbed latents to provide adversarial loss (GAN loss).
• Efficiency: The student model integrates Classifier-Free Guidance (CFG) directly, allowing for single-forward-pass generation per step.

Key Contributions

1. Unified Framework: Introduction of AudioX-Turbo, a single model capable of generating both sound effects and music from flexible combinations of text, video, and audio inputs, relaxing the constraints of task-specific systems.
2. Large-Scale Dataset: The creation of IF-caps-Pro (~9.2M samples) via a novel two-stage curation and annotation pipeline, providing the necessary fine-grained supervision for unified multimodal training.
3. Efficient Inference: A distillation strategy that reduces inference cost by up to 25× (achieving comparable quality to multi-step baselines with only 4 sampling steps) while maintaining strong instruction-following capabilities through the use of DMD and a diffusion-based discriminator.
4. Cross-Modal Regularization: The discovery that improving the quality and granularity of textual supervision (via the annotation pipeline) leads to better modality alignment, jointly boosting performance across all conditioning modalities (text, video, and audio).

Results

Extensive benchmarking demonstrates that AudioX-Turbo achieves state-of-the-art (SOTA) or highly competitive performance across multiple tasks:

• Quality: On benchmarks like AudioCaps, VGGSound, MusicCaps, and V2M-bench, the model outperforms or matches specialized SOTA models in metrics such as Inception Score (IS), Fréchet Audio Distance (FAD), and alignment scores (CLAP, ImageBind AV).
• Instruction Following: On the newly constructed T2A-bench and AudioTime, AudioX-Turbo significantly outperforms baselines in category, count, and ordering accuracy, demonstrating superior fine-grained control.
• Efficiency: Using only 4 sampling steps, AudioX-Turbo matches the performance of its multi-step teacher (AudioX-Base) and significantly outperforms multi-step baselines forced to run at 4 steps. It reduces the number of function evaluations (NFE) by approximately 25× compared to standard 50-step baselines while maintaining real-time factors (RTF) well below 1.0.
• Ablation Studies: Confirm the necessity of the MAF module for multimodal fusion, the effectiveness of the two-stage data curation pipeline (specifically the "GeminiCap-aug" strategy), and the importance of the adversarial objective for perceptual fidelity.

Significance

The paper claims that AudioX-Turbo represents a significant step toward practical, real-time "anything-to-audio" generation. By unifying diverse modalities within a single efficient framework, it addresses the fragmentation of current audio generation research. The work demonstrates that high-fidelity, instruction-following audio generation is achievable with drastically reduced computational costs, making it viable for latency-sensitive applications. Furthermore, the proposed dataset and the observed cross-modal regularization effect provide empirical insights and resources for future multimodal audio generation research.

Limitations and Future Work: The authors acknowledge that the current model is trained on short 10-second clips, limiting long-form generation (e.g., full film scores). Additionally, the framework currently excludes speech generation, and instruction-following accuracy may degrade under extreme conditions (e.g., many concurrent sound events). Future work aims to address long-context modeling, unify speech generation, and explore flexible-step generation.