How Should LLMs Listen While Speaking? A Study of User-Stream Routing in Full-Duplex Spoken Dialogue

Technical Summary: How Should LLMs Listen While Speaking? A Study of User-Stream Routing in Full-Duplex Spoken Dialogue

Problem Statement
Full-duplex spoken dialogue, where a user and a system can speak and listen simultaneously, is essential for natural human-machine interaction. However, standard Large Language Models (LLMs) are pre-trained to extend a single, coherent token sequence and do not natively support processing external user input while generating a response. This creates a critical architectural challenge: how should the incoming user speech stream be routed into the LLM during ongoing generation? Existing approaches either interleave streams (increasing sequence length and latency) or fuse channels (risking context corruption when streams overlap). The specific failure mode of "context corruption"—where overlapping user speech interferes with the model's generation context if it fails to stop in time—has not been systematically characterized or compared against alternative routing strategies.

Methodology
The authors propose a unified framework to extend a text-only LLM into a full-duplex spoken dialogue system, enabling a controlled comparison of two distinct user-stream routing strategies under a shared training pipeline:

1. System Architecture: The system comprises a streaming speech encoder (adapted from Whisper), a speech adapter, a text-only LLM backbone (Qwen3-1.7B), an audio decoder (audio head), and a user-stream routing module. The system operates on a shared duplex timeline with special tokens for waiting (<USER_WAIT>, <TEXT_WAIT>, <AUDIO_WAIT>) and interruption (<TEXT_INT>, <AUDIO_INT>).
2. Routing Strategies:
   • Channel Fusion (CF-Duplex): The user stream embeddings are directly fused with the model's text and audio streams at each time step using a gating mechanism (sigmoid gate + MLP). This provides direct, time-aligned access to user input but merges the user stream into the same context used for generation.
   • Cross-Attention Routing (XA-Duplex): The user stream is maintained as a separate external memory (keys and values). The LLM accesses this memory via cross-attention adapters (similar to Flamingo) while keeping its native autoregressive generation context separate.
3. Training Curriculum: Both models undergo a three-stage training process:
   • Stage 1: ASR and streaming TTS.
   • Stage 2: Turn-based spoken dialogue (S2TD and S2TSD).
   • Stage 3: Full-duplex dialogue training, explicitly simulating user interruptions and backchannels with dynamic overlap ranges.
4. Data Construction: The study utilizes a massive dataset (217k hours of ASR, 104k hours of TTS) and constructs full-duplex examples by simulating interruptions (context-dependent and independent) and backchannels on top of turn-based dialogues.

Key Results
Experiments on spoken question answering (QA) and full-duplex interaction benchmarks (Full-Duplex-Bench v1.0 & v1.5) reveal a clear tradeoff between the two routing strategies:

• Semantic Grounding vs. Context Robustness:
  • CF-Duplex achieves significantly stronger performance in spoken question answering across all datasets (LLaMAQ, TriviaQ, WebQ, AlpacaEval). It demonstrates better semantic grounding from speech input.
  • XA-Duplex underperforms in QA tasks but exhibits superior robustness in handling overlapping speech.
• Interruption Handling:
  • On Full-Duplex-Bench, CF-Duplex achieves state-of-the-art or competitive results in takeover rate (TOR) and response quality.
  • However, qualitative analysis of "missed interruption" cases (where the model fails to stop speaking) shows a critical divergence. When CF-Duplex fails to stop, the fused user stream corrupts the generation context, leading to semantically incoherent continuations that blend the interruption with the ongoing response.
  • In contrast, when XA-Duplex fails to stop, it largely preserves the coherence of its original response, as the user stream remains in a separate memory and does not contaminate the generation context.
• Training Dynamics: The performance gap emerges early in Stage 2 (dialogue training), where CF-Duplex consistently outperforms XA-Duplex in semantic tasks, while XA-Duplex maintains better structural separation.

Significance and Claims
The paper establishes user-stream routing as a central design axis for full-duplex spoken dialogue. The authors claim that the choice between channel fusion and cross-attention routing is not merely an implementation detail but a fundamental architectural decision that dictates the tradeoff between:

1. Semantic Integration: The ability to deeply understand and respond to user input (favored by Channel Fusion).
2. Context Robustness: The ability to maintain coherent generation states when the model fails to detect an interruption (favored by Cross-Attention).

The study provides practical guidance: if the priority is high-quality semantic understanding and the system can rely on robust interruption detection, channel fusion is preferable. If the priority is ensuring that generation remains coherent even when interruption detection fails, cross-attention routing offers a safer, more robust alternative. The work highlights that current full-duplex systems often implicitly choose one strategy over the other without understanding these specific failure modes.

Limitations
The authors note that the study is limited to two routing strategies within a single framework and a specific model scale (1.7B backbone). The observed tradeoffs may evolve with larger model scales or different architectural designs, which requires further investigation.