Joint Speech and Text Training for LLM-Based End-to-End Spoken Dialogue State Tracking

Technical Summary: Joint Speech And Text Training For LLM-based End-To-End Spoken Dialogue State Tracking

Problem Statement
End-to-end (E2E) spoken Dialogue State Tracking (DST) faces a significant bottleneck: the scarcity of annotated spoken dialogue data required to train robust models across diverse domains. While cascading Automatic Speech Recognition (ASR) with text-based DST modules is a traditional approach, it suffers from error propagation and latency. Recent E2E methods that directly predict dialogue states from speech using Large Language Models (LLMs) and speech encoders have shown promise but struggle to generalize to new domains without specific spoken training data for those domains. Collecting such domain-specific spoken data is costly and labor-intensive. Conversely, textual DST data is abundant and easier to acquire, yet standard E2E speech models cannot directly leverage unpaired text data to improve performance on spoken inputs.

Methodology
The authors propose a Joint Speech and Text E2E DST framework designed to enable cross-domain generalization without requiring spoken training data from the target domains. The core architecture builds upon a baseline E2E model (comprising a speech encoder, a connector, and an LLM) by augmenting it with a text encoder.

• Architecture: The model accepts two types of inputs during training:
  1. Speech Pipeline: User speech is encoded by a pretrained speech encoder (e.g., WavLM), passed through a connector, and prepended to the dialogue history embeddings before entering the LLM.
  2. Text Pipeline: Natural language user queries are encoded by a dedicated text encoder (a Transformer with dimensions matching the connector), passed through the same connector, and fed into the LLM.
• Training Strategy: The system employs a two-phase training process:
  1. ASR Pretraining: The speech encoder and connector are pretrained on diverse speech data using a cross-entropy objective, with the LLM frozen.
  2. Joint Finetuning: The text encoder is introduced, and LoRA (Low-Rank Adaptation) layers are inserted into the LLM. The connector, text encoder, and LoRA parameters are jointly finetuned.
• Loss Function: At each training step, the model processes a batch of speech and a batch of text. The optimization minimizes the sum of three cross-entropy losses:
  1. DST loss on spoken data.
  2. DST loss on unpaired textual data.
  3. DST loss on the transcription of the speech batch (to ensure the text encoder aids the speech pipeline's understanding of the target domain).
• Inference: Crucially, the text encoder is discarded during inference. The model operates solely on speech input, incurring no additional inference latency compared to the baseline.

Key Contributions and Results
The paper presents experiments on the SpokenWOZ and Speech-aware MultiWOZ datasets, demonstrating the efficacy of the proposed method:

• Cross-Domain Generalization: The method successfully adapts models trained on speech from a source domain to perform DST on target domains where only textual data is available. For instance, training on SpokenWOZ speech combined with MultiWOZ text significantly improved Joint Goal Accuracy (JGA) on the MultiWOZ validation set compared to a baseline trained only on SpokenWOZ speech.
• Comparison with Alternatives: The authors compared their approach to a strategy using Text-To-Speech (TTS) to synthesize spoken data from text. They found that joint text training achieved comparable results to the TTS approach but without the added complexity and dependency on high-quality TTS systems, which is particularly advantageous for low-resource languages.
• Mechanism of Improvement: Ablation studies revealed that improvements stem from two factors: (1) exposing the LLM to target domain dialogue states via text, and (2) the text encoder actively helping the speech pipeline learn to extract these states from natural language. Simply finetuning LoRA on text without the text encoder yielded lower performance than the full joint approach.
• Scalability: The benefits of joint text training were observed to be more pronounced with larger LLMs (e.g., Gemma-3-12B-it). In some cases, the larger models trained with joint text achieved performance on target domains nearly identical to models trained on paired speech data from those same domains.
• Slot vs. Value Recognition: Analysis indicated that while adding text from a different city (e.g., MultiWOZ New York data to Cambridge-based SpokenWOZ training) improved slot key recognition, it did not necessarily improve slot value recognition (e.g., specific place names), highlighting the distinction between structural and content-based generalization.

Significance
The paper claims that this work provides a practical solution to the data scarcity problem in spoken DST. By leveraging the abundance of unpaired textual DST data, the proposed method enables the creation of multi-domain spoken dialogue systems without the prohibitive cost of collecting spoken data for every target domain. The authors emphasize that this approach allows for the application of established text-based DST enhancement strategies (such as slot augmentation and paraphrasing) to spoken systems indirectly, opening new avenues for research in low-resource and multi-domain spoken dialogue scenarios.