Towards Efficient Speech-Text Jointly Decoding within One Speech Language Model

Technical Summary: Towards Efficient Speech-Text Jointly Decoding within One Speech Language Model

1. Problem Statement

As Speech Language Models (Speech LMs) evolve toward end-to-end systems capable of generating both text and speech, the decoding paradigm—the method by which the model produces and consumes these two modalities—becomes a critical bottleneck. The researchers identify three primary challenges:

• Performance vs. Efficiency Trade-off: Existing interleaved decoding (alternating text and speech tokens) provides high alignment but suffers from extremely long sequence lengths, leading to slow inference.
• Lack of Fair Comparison: Previous studies often compare different decoding strategies using different base models, tokenizers, or datasets, making it difficult to isolate the impact of the decoding paradigm itself.
• Alignment Quality: Ensuring that the generated speech accurately reflects the semantic content of the generated text (speech-text alignment) is difficult in parallel or decoupled architectures.

2. Methodology

The authors conduct a systematic investigation using a controlled experimental setup, keeping the base model (Phi4-MM), speech tokenizer (S3Tokenizer), and training data constant to ensure a fair comparison.

A. Comparison of Decoding Paradigms:
They evaluate three representative strategies:

1. Interleaved: Alternates between text and speech tokens in a single sequence.
2. Parallel: Predicts one text token and multiple speech tokens in a single forward pass, averaging their embeddings for the next step.
3. Thinker-Talker: A decoupled approach where an LLM (Thinker) generates text, and a separate model (Talker) generates speech based on the Thinker's hidden states.

B. Proposed Innovation: Early-Stop Interleaved (ESI) Pattern:
To solve the inefficiency of the interleaved approach, the authors propose the ESI paradigm. In standard interleaved decoding, "padding" text tokens are often required to maintain a fixed text-to-speech ratio once text is exhausted. ESI introduces a special token (<S>) following the text End-of-Sentence (EOS) token. Once <S> is predicted, the model stops generating text tokens entirely and switches to generating only speech tokens. This effectively eliminates redundant padding.

C. Data Curation for Speech QA:
To improve reasoning, they curated high-quality Speech Question Answering (QA) datasets by:

• Rewriting short text answers into conversational full sentences via LLM APIs.
• Synthesizing audio using zero-shot TTS with high speaker diversity.
• Filtering data using ASR (Automatic Speech Recognition) to ensure a Word Error Rate (WER) below 20%.

3. Key Contributions

• Systematic Benchmarking: Provided the first fair, controlled comparison of joint speech-text decoding strategies within a single LM.
• ESI Paradigm: Introduced a novel decoding mechanism that reduces total sequence length by approximately 25%, significantly accelerating inference while maintaining or slightly improving performance.
• Data Pipeline: Developed a robust pipeline for generating high-quality, conversational speech-in/speech-out QA datasets.

4. Results

• Paradigm Superiority: The Interleaved pattern consistently outperformed the Parallel pattern in alignment (Rel. score) and accuracy. The Parallel pattern showed significantly higher WER, indicating poor speech-text synchronization.
• ESI Efficiency and Performance: The ESI pattern achieved comparable or even better performance than standard interleaving. For example, on the Web Questions benchmark, the 7B ESI model improved speech-out accuracy compared to the standard interleaved version.
• State-of-the-Art (SOTA) Comparison:
  • The Ours 7B-ESI model outperformed LLaMA-Omni2-14B (a much larger model) in speech-out accuracy and alignment.
  • Even the smaller 3.8B-ESI variant outperformed GLM-4-Voice (9B) and Moshi (7B) on several metrics.
• Data Impact: Ablation studies confirmed that adding curated QA datasets (Natural Questions and TriviaQA) to the base VoiceAssistant data leads to significant gains in model intelligence and accuracy.

5. Significance

This work provides a blueprint for developing efficient, real-time conversational AI. By proving that the interleaved paradigm is superior for alignment but can be optimized via the ESI pattern, the authors offer a practical solution to the latency issues that have previously hindered end-to-end speech LMs. The research highlights that for spoken dialogue systems, the synergy between the decoding architecture and high-quality conversational data curation is just as vital as the scale of the underlying language model.