⚡ electrical engineering

Unified Audio Intelligence Without Regressing on Text Intelligence

The paper introduces Audex, a unified 30B audio-text LLM built on a strong text-only MoE backbone that achieves state-of-the-art performance across diverse audio and speech tasks while preserving the original model's advanced reasoning and agentic capabilities without significant regression.

Original authors: Zhifeng Kong, Sang-gil Lee, Jaehyeon Kim, Boxin Wang, Zihan Liu, Sungwon Kim, Yang Chen, Arushi Goel, Rajarshi Roy, Wenliang Dai, Zhuolin Yang, Yangyi Chen, Dongfu Jiang, Sreyan Ghosh, Tuomas Rintamak

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhifeng Kong, Sang-gil Lee, Jaehyeon Kim, Boxin Wang, Zihan Liu, Sungwon Kim, Yang Chen, Arushi Goel, Rajarshi Roy, Wenliang Dai, Zhuolin Yang, Yangyi Chen, Dongfu Jiang, Sreyan Ghosh, Tuomas Rintamaki, Andrew Tao, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: A "Swiss Army Knife" Brain That Doesn't Forget How to Think

Imagine you have a brilliant professor who is a master at writing essays, solving complex math problems, and coding software. This professor is your text-only AI. Now, imagine you want to teach this professor to also understand music, recognize bird songs, and speak back to you in a human voice.

Usually, when you teach a genius a new, difficult skill (like juggling), they might get a little clumsy at their original job (like writing). They might start making grammar mistakes or forgetting facts because their brain is so busy learning the new trick.

This paper introduces "Audex," a new AI model that learns to juggle audio and speech without dropping its ability to think clearly.

The researchers built Audex on top of a very smart text-only brain called Nemotron-Cascade-2. Their goal was simple: Make an AI that can hear, speak, and understand sound, but never lose its ability to reason, solve math problems, or follow instructions.

How It Works: The "Universal Translator" Trick

Most AI models that handle sound are like two separate people holding hands: one person listens, and another person speaks. If they don't talk to each other perfectly, the result is messy.

Audex is different. It is a single, unified brain. Here is how it manages to do everything at once:

The Ear (Audio Encoder): When Audex hears a sound (like a dog barking or someone speaking), it doesn't try to understand the raw sound waves immediately. Instead, it uses a specialized "translator" (an audio encoder) to turn the sound into a list of numbers that look just like the words the AI already knows.
The Brain (The LLM): These numbers are fed into the main brain. Because the brain sees them as "tokens" (like words), it treats a sound clip exactly the same way it treats a sentence of text.
The Mouth (Audio Codec): When Audex wants to speak or make a sound, it doesn't just "think" the words; it predicts the next "sound token" in the sequence, just like it predicts the next word in a sentence.

The Analogy: Imagine a chef who usually only cooks with text recipes. Audex is like teaching that chef to cook with sound ingredients. Instead of needing a separate kitchen for sound, Audex just translates the sound ingredients into the same language the chef already speaks. The chef can now cook a "sound soup" without forgetting how to bake a "text cake."

The Secret Sauce: Training Without Breaking the Brain

The researchers faced a big challenge: If you train a smart text AI too hard on audio, it might forget how to be smart. To prevent this, they used a careful, step-by-step training recipe:

Warm-up: First, they taught the AI how to handle the "sound ingredients" (the audio tokens) without changing its core knowledge.
Layered Learning: They didn't throw everything at the AI at once. They taught it to generate sound first, then taught it to understand sound, and finally mixed it all together. This is like learning to ride a bike with training wheels before trying to ride a unicycle.
The "No-Regression" Promise: After all the training, they tested the AI on its old skills (math, logic, coding). The results were shocking: Audex got just as good at these tasks as the original text-only version. It didn't lose any of its "genius" while gaining the ability to hear and speak.

What Can Audex Actually Do?

According to the paper, Audex is a "Swiss Army Knife" for audio. It can:

Listen and Understand: It can answer questions about what it hears (e.g., "Is that a dog or a wolf?" or "What is the mood of this music?").
Transcribe and Translate: It can turn spoken words into text and translate them into other languages, even in noisy rooms.
Speak and Sing: It can take a text prompt and generate human-like speech or even create music and sound effects (like "rain falling" or "birds chirping").
Talk to Talk: It can take a voice input and generate a different voice output (speech-to-speech).

The Results: The Best of Both Worlds

The paper compares Audex to other top-tier AI models.

On Text: It performs as well as the smartest text-only models, solving complex math and logic puzzles without a drop in performance.
On Audio: It beats many other models at recognizing speech and understanding general sounds.
The "Thinking" Mode: Unlike some models that get "dumber" when they try to think and speak at the same time, Audex can "think" (reason) about a sound problem and then generate the answer in audio, all in one go.

In Summary

This paper presents Audex, a model that proves you don't have to choose between being a "smart thinker" and a "good listener/speaker." By treating sound as just another type of language, the researchers built a single AI that can hear, speak, and reason with equal brilliance, keeping its original intelligence intact while mastering the world of sound.

Technical Summary: Unified Audio Intelligence Without Regressing on Text Intelligence

Problem Statement

The development of Artificial General Intelligence (AGI) requires models capable of understanding, reasoning about, and generating audio and speech. While recent efforts have produced audio Large Language Models (LLMs) for understanding and generation, a critical challenge remains: multimodal regression. Models capable of multimodal inputs and outputs often exhibit significant degradation in text-based reasoning, knowledge, and alignment compared to their text-only counterparts. Existing open-source models often lack multimodal generation capabilities, while proprietary models (e.g., GPT-4o, Gemini 2.5) integrate these features but do not release weights. Furthermore, existing unified models often struggle to maintain the sophisticated reasoning and agentic capabilities of their text-only backbones when audio generation is added.

Methodology

Model Architecture

The paper introduces Nemotron-Labs-Audex-30B-A3B (Audex), a unified audio-text LLM built upon the Nemotron-Cascade-2-30B-A3B, a strong text-only Mixture-of-Experts (MoE) LLM based on a hybrid Mamba2-Transformer architecture (30B total parameters, 3B activated).

Unified Design: Audex utilizes a single Transformer decoder. Audio inputs are processed by an Audio Encoder (finetuned AF-Whisper) and projected into the text embedding space via MLP adapters.
Tokenization: The vocabulary is extended from 131,072 to 205,312 tokens to accommodate discrete audio tokens.
- Speech: Tokenized using X-Codec2 (50 Hz, single-layer FSQ, 65,536 vocabulary).
- Non-Speech Audio: Tokenized using X-Codec (50 Hz, 4-layer RVQ, 4,096 vocabulary).
- Generation: The model predicts text tokens and quantized audio tokens uniformly using standard cross-entropy loss. Special tokens (e.g., <speechgen_start>, <audiogen_end>) manage generation boundaries.
Decoding: For non-speech audio, an enhancement VAE (based on BigVGAN-v2) improves quality. For speech, a streaming ConvNeXt decoder replaces the non-causal X-Codec2 decoder for faster inference.

Training Strategy

The training pipeline involves meticulous data curation (157.4B audio tokens, 320.5B text tokens) and a multi-stage approach:

Supervised Fine-Tuning (SFT): The authors compare two strategies:
- Multi-stage SFT: A curriculum starting with text-only SFT, followed by an "Audio Warmup" (training only adapters/embeddings), then "Audio Generation SFT," and finally integrating "Audio Understanding" and ASR/AST tasks. This approach preserves long-context attention mechanisms.
- Single-stage Consolidated SFT: Combines all data types in one stage. While cheaper, this was found to degrade long-context performance (NIAH benchmark) significantly.
- Decision: The Multi-stage SFT strategy was selected for the final model to maintain text reasoning capabilities.
Reinforcement Learning (RL): The model undergoes text-only Cascade RL and Multi-domain On-policy Distillation (MOPD), mirroring the training of the text-only backbone. This stage improves text reasoning and alignment without causing catastrophic forgetting of audio capabilities.
Classifier-Free Guidance (CFG): To improve generation quality, the model is trained with unconditional samples (text replaced by padding tokens). At inference, CFG is applied with specific hyperparameters: $\lambda \approx 3$ for Text-to-Audio (TTA) and $\lambda \approx 1.5$ for Text-to-Speech (TTS).

Key Contributions

Unified Architecture without Regression: Audex achieves state-of-the-art (SOTA) performance in audio understanding, speech recognition, translation, TTS, and audio generation while preserving the text intelligence of its 30B MoE backbone. Unlike other unified models, it shows marginal or no regression on reasoning, knowledge, and agentic benchmarks.
Open Release: The authors release model checkpoints for both the 30B-A3B model and a smaller 2B variant (Audex-2B), facilitating open research.
General Audio Generation: Audex is the only strong open model demonstrated to support general audio generation (non-speech) alongside speech, unifying text-to-audio and text-to-speech within a single autoregressive LLM framework.
Training Recipe: The paper details a specific curriculum (Multi-stage SFT followed by text-only RL) that successfully integrates complex audio generation tasks into a reasoning-capable LLM without breaking its long-context or reasoning abilities.

Results

Text Capabilities

Audex-30B-A3B matches or slightly exceeds its text-only backbone (Nemotron-Cascade-2) on key benchmarks:

Reasoning: Achieves 91.2% on AIME 2025 and 92.2% on HMMT Feb25, outperforming Qwen3-Omni-30B-A3B-Thinking which showed significant degradation.
Long Context: Maintains high performance on NIAH (99.4% at 256K, 83.4% at 1M), whereas single-stage SFT models dropped to near zero.
Agentic & Alignment: Scores 57.2 on $\tau^2$ -Bench and 81.6 on ArenaHard v2, comparable to the text-only baseline.

Audio Capabilities

Speech Recognition: Achieves SOTA among open models on OpenASR (6.82% WER average) and noisy LibriSpeech, outperforming Step-Audio-R1.1 and Qwen3-Omni-30B-A3B-Thinking.
Audio Understanding: Matches strong baselines on MMAU (75.6) and demonstrates superior trustworthiness on Audio Entailment (95.0) and CMM Hallucination detection.
Generation:
- TTS: Achieves 1.70 WER on Seed-TTS-Eval (fixed voice).
- TTA: Achieves SOTA results among public-data models on AudioCaps (66.9 OpenL3 FD) and SongDescriber (62.7 OpenL3 FD), outperforming diffusion baselines like AudioLDM2 and Stable Audio Open.

Significance

The paper claims that Audex represents a critical step toward AGI by demonstrating that native multimodal LLMs capable of both understanding and generation can be built without compromising text intelligence. By successfully integrating audio generation into a high-reasoning MoE backbone through a carefully curated multi-stage training recipe, Audex bridges the gap between specialized audio models and general-purpose reasoning engines. The authors emphasize that their approach avoids the "regression" seen in other unified models, proving that audio capabilities can be added to frontier LLMs while retaining their core reasoning, knowledge, and agentic strengths.