⚡ electrical engineering

Scaling Open Discrete Audio Foundation Models with Interleaved Semantic, Acoustic, and Text Tokens

This paper introduces SODA, a suite of native audio foundation models that jointly model semantic, acoustic, and text tokens, and establishes a validated training recipe and scaling laws through an extensive empirical study of 64 models to demonstrate superior performance across diverse audio and cross-modal tasks.

Original authors: Potsawee Manakul, Woody Haosheng Gan, Martijn Bartelds, Guangzhi Sun, William Held, Diyi Yang

Published 2026-02-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Potsawee Manakul, Woody Haosheng Gan, Martijn Bartelds, Guangzhi Sun, William Held, Diyi Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot to understand and speak like a human. For a long time, scientists tried two main ways to do this, but both had a major flaw:

The "Translator" Approach: They took a super-smart text robot (like a chatbot) and tried to bolt a microphone onto it. The problem? The robot was so focused on the meaning of words that it forgot how to sound natural. It sounded like a robot reading a script, missing the emotion, the breath, and the unique voice of the speaker.
The "Mime" Approach: They built a robot that only listened to sound patterns but ignored the actual words. It could mimic a voice perfectly but couldn't understand what was being said or hold a conversation.

This paper introduces a new kind of robot called SODA (Scaling Open Discrete Audio). Think of SODA as a "Universal Audio Chef" that learns to cook with three different ingredients at the same time: Meaning (what is said), Sound (how it sounds), and Text (the written words).

Here is how they did it, explained simply:

1. The Secret Sauce: Mixing the Ingredients

The researchers realized that to make a truly smart audio robot, you can't just feed it one type of data. You have to mix them up, sentence by sentence.

The Old Way: Feed the robot a book, then feed it a recording.
The SODA Way: Feed it a sentence of text, then immediately the sound of that sentence, then the next sentence, then the next sound. It's like teaching a child by showing them a picture of a dog, saying "dog," and then playing a bark, all in one continuous stream.

They found the perfect recipe:

95% Audio: Real human speech (from YouTube and audiobooks).
5% Text: High-quality written stories.
The Magic Ratio: They discovered that adding just a tiny bit of text (5%) helped the robot understand what people were saying without ruining its ability to understand how they sounded.

2. The "Goldilocks" Rule (Scaling Laws)

In the world of AI, there's a famous rule called "Chinchilla" for text robots: "If you double the brain size, you should double the amount of reading material."

The researchers asked: "Does this rule work for sound?"

They built 64 different versions of their audio robot, ranging from tiny to huge, and tested them with different amounts of data. They found that sound is "denser" with information than text.

The Analogy: Imagine text is like a clear, slow-moving river. Sound is like a fast, churning waterfall. You need a much bigger bucket (more data) to catch the same amount of water (information) from the waterfall.
The Discovery: To get the best audio robot, you need to increase the data much faster than you increase the brain size. Specifically, for every step up in brain size, you need about 1.6 times more data.

3. Starting from Scratch vs. Copying a Friend

Many AI researchers start by taking a pre-trained text robot and trying to teach it audio. The paper calls this "Warm-Starting."

The Result: It was messy. The robot got confused, had "tantrums" (training errors), and actually got worse at understanding speech.
The Better Way: They started with a blank slate ("Cold-Start"). They built the robot from zero, learning audio and text together from the very first second.
The Analogy: It's like trying to teach a professional pianist to play the drums. They might keep trying to play piano chords on the drum kit. It's better to start with a fresh student who learns drums and piano simultaneously from day one.

4. What Can SODA Do?

Because they taught it this "Universal" way, SODA is incredibly flexible. You don't need to build a new robot for every task. You just give it a new instruction, and it adapts.

Speech-to-Text: It can listen to a messy recording and write it down perfectly.
Text-to-Speech: It can read a text and speak it in a natural, human voice.
Voice Translation: This is the coolest trick. You can speak Spanish, and SODA can translate it to English while keeping your original voice. It's like a translator who speaks your exact tone and style, not just the words.

The Big Takeaway

This paper proves that if you treat audio, speech, and text as one big, mixed-up puzzle rather than separate subjects, you can build a much smarter, more flexible AI.

They released their "recipe," their "ingredients," and the "kitchen" (the code and models) for everyone to use. This means other scientists can now build their own audio robots without having to reinvent the wheel, potentially leading to better voice assistants, more accessible tools for the deaf or hard of hearing, and more natural human-computer interactions.

In short: They stopped trying to force a text brain to hear, and instead built a brain that was born to listen, speak, and read all at once.

1. Problem Statement

Current audio language models face significant limitations in achieving general-purpose audio understanding and generation:

LLM-Centric Architectures: Models like SALMONN or Qwen3-Omni extend pre-trained text LLMs with audio encoders. They suffer from a "semantic bottleneck," where fine-grained acoustic details are lost during compression, limiting high-fidelity audio-to-audio modeling.
Semantic-Only Models: Approaches like SpiritLM or TWIST use discrete semantic tokens (e.g., HuBERT units) but discard acoustic details, preventing high-fidelity generation and detailed acoustic understanding.
Native Audio Models: Models like Moshi or Llama-Mimi model acoustic tokens directly but often lack text integration or systematic scaling studies, limiting their cross-modal capabilities (e.g., ASR, TTS).
Lack of Scaling Laws: Unlike text LLMs, where scaling laws (e.g., Chinchilla) dictate optimal compute allocation, there is no established understanding of how model size ( $N$ ) and training data ( $D$ ) should scale for discrete audio models, particularly given the lower information density of audio tokens compared to text.

2. Methodology

The authors propose SODA (Scaling Open Discrete Audio), a suite of native foundation models trained using a unified next-token prediction (NTP) framework.

A. Model Architecture & Tokenization

Architecture: Decoder-only Transformers based on the Qwen3 architecture (incorporating QK-Norm for stability), trained from scratch (cold-start).
Tokenization: Uses the Mimi neural codec (12.5 Hz, 8 RVQ codebooks).
- Codebook 1: Captures semantic content.
- Codebooks 2–8: Capture acoustic details.
Interleaving Strategy: The core innovation is utterance-level interleaving of tokens. Instead of word-level alignment, the model processes sequences where audio tokens (semantic + acoustic) and text tokens (transcripts) are interleaved within a single utterance. This supports four capabilities in one model:
1. Audio continuation
2. Text continuation
3. Audio $\to$ Text (ASR)
4. Text $\to$ Audio (TTS)

B. Training Data & Recipe

Data Sources:
- Speech: Yodas (YouTube-based, diverse styles) and Emilia (spontaneous speech). MLS was excluded due to transcript mismatches and lack of length diversity.
- Text: Nemotron-CC (high-quality web corpus).
Optimal Mixture: Through ablation, the authors determined a 5% Text / 95% Speech mixture maximizes text knowledge without degrading audio performance.
Token Composition: The optimal configuration includes Semantic + Acoustic + Text tokens. While adding acoustic tokens slightly reduces semantic understanding scores, it is essential for high-fidelity generation and cross-modal tasks.

C. Scaling Law Analysis (IsoFLOP Study)

The authors conducted the first large-scale scaling study for discrete audio models:

Scope: Trained 64 models across 7 compute budgets ranging from $3 \times 10^{18}$ to $3 \times 10^{20}$ FLOPs.
Metric Validation: Established that Validation Negative Log-Likelihood (NLL) on interleaved audio+text data is a highly reliable predictor of downstream performance (ASR, TTS, semantic understanding).
Scaling Laws: Derived optimal scaling exponents for compute $C$ $C$ :
- Optimal Model Size: $N^* \propto C^{0.367}$
- Optimal Data Size: $D^* \propto C^{0.579}$
- Key Insight: Data must scale 1.6 $\times$ faster than model size. This contrasts with the Chinchilla law ( $N^*, D^* \propto C^{0.5}$ ) and suggests discrete audio tokens have lower information density than text tokens, requiring more data to achieve effective learning.

3. Key Contributions

Unified Native Audio Foundation Model: Demonstrated that a single decoder-only model can jointly model semantic content, acoustic details, and text, enabling diverse tasks (ASR, TTS, S2ST, continuation) without specialized encoders/decoders.
Validated Training Recipe: Established a robust recipe involving utterance-level interleaving, a 5% text mixture, and specific data sources (Yodas + Emilia).
First Scaling Laws for Discrete Audio: Provided empirical evidence that data should scale significantly faster than model size for audio models ( $D^* \propto C^{0.579}$ ), challenging the assumption that text scaling laws apply directly to audio.
Cold-Start vs. Warm-Start: Showed that cold-start (training from scratch) is superior for audio tasks, offering better training stability and ASR performance, whereas warm-starting from text LLMs leads to instability and fails to fully recover text knowledge.
SODA Model Suite: Released a family of models (135M to 4B parameters) trained on 500B tokens, serving as a flexible backbone for future research.

4. Key Results

Performance: SODA-4B achieves competitive results across benchmarks:
- ASR: 5.0% WER (LibriSpeech-clean).
- TTS: 6.1% WER and 0.56 Speaker Similarity.
- Acoustic Understanding: ~70% accuracy on Salmon.
- Text Knowledge: Significant emergence in HellaSwag (52.6% accuracy) and tWUGGY (74.8%).
Scaling Behavior:
- Cross-modal & Acoustic Skills: Show signs of saturation at larger scales (diminishing returns).
- Semantic & Text Knowledge: Exhibit emergent behavior, with performance accelerating rapidly as model size and data increase beyond the IsoFLOP regime.
Fine-Tuning Flexibility: SODA was successfully fine-tuned for Voice-Preserving Speech-to-Speech Translation (S2ST) using the same architecture and NTP objective, achieving state-of-the-art speaker similarity (~0.47) without specialized architectures.

5. Significance

This work bridges the gap between text-centric LLMs and specialized audio models. By proving that native audio foundation models can be scaled effectively using next-token prediction, it offers a more flexible and unified approach to multimodal AI. The discovery that data scales faster than model size for audio provides a critical guideline for future resource allocation in audio foundation model training. Furthermore, the release of open models, data, and training recipes democratizes access to high-fidelity audio generation and understanding research.