Discourse-Aware Dual-Track Streaming Response for Low-Latency Spoken Dialogue Systems

1. Problem Statement

Current cascaded Spoken Dialogue Systems (SDS), which follow a sequential ASR (Automatic Speech Recognition) $\to$ LLM (Large Language Model) $\to$ TTS (Text-to-Speech) pipeline, suffer from high response latency.

• The Bottleneck: Conventional systems must wait for the complete transcription of the user's speech and the full semantic reasoning of the LLM before initiating speech synthesis. This "wait-to-think" paradigm results in response onsets often exceeding one second, disrupting natural turn-taking (which humans achieve in sub-300ms).
• The Dilemma: While end-to-end speech models offer lower latency, they are computationally expensive and lack modularity. Conversely, cascaded systems offer modularity and controllability but struggle with the sequential accumulation of delays.
• The Gap: Existing incremental generation methods still rely on waiting for sufficient semantic content before speaking, failing to exploit the human ability to "speak while thinking" using low-risk discourse cues.

2. Methodology: The DDTSR Framework

The authors propose DDTSR, a low-latency architecture that decouples early, minimal-committal speech generation from deep, knowledge-intensive reasoning. It operates on three core mechanisms:

A. Connective-Guided Small–Large Model Synergy

Instead of a single monolithic model, DDTSR decomposes response generation into two parallel tracks:

1. Lightweight Model ($f_s$): Dedicated to generating discourse connectives (e.g., "Well," "I see," "Hmm"). These are semantically safe, require minimal reasoning, and serve as "fillers" to hold the conversational floor.
2. Large Model ($f_l$): Performs deep, knowledge-intensive reasoning to generate the substantive answer.

• Mechanism: Both models operate concurrently upon receiving user input. The lightweight model emits connectives immediately, while the large model works in the background. The final output is a temporal concatenation: $y = \text{TTS}(f_s(u)) \oplus \text{TTS}(f_l(u))$.

B. Streaming-Based Cross-Modal Collaboration

DDTSR restructures the temporal workflow to overlap processing stages:

1. Early Emission during Streaming ASR: The system does not wait for the final ASR transcript. Instead, it processes partial hypotheses ($\hat{u}^{(k)}$) from a streaming ASR.
2. Commitment Policy: The lightweight model analyzes partial transcripts to predict a connective and a binary "commit signal." Once a reliable commit point is reached (often while the user is still speaking), TTS is triggered immediately to synthesize the connective.
3. Parallel Execution: While the connective audio plays, the large model continues reasoning on the finalized transcript. The system seamlessly transitions from the connective audio to the substantive response audio without silence gaps.

C. Curriculum-Learning-Based Discourse Continuity Enhancement

To ensure the early connectives do not contradict the final answer, the system employs a specialized training strategy:

• Joint Training Objectives:
  • Style Consistency Loss: Aligns the connective with the semantic style of the anticipated large-model response.
  • Coherence Loss: Minimizes the perplexity gap between the connective-response pair generated by the small model and the original distribution, ensuring logical flow.
  • Prior Regularization: Prevents overfitting and preserves natural connective diversity.
• Curriculum Learning: The model is trained progressively from full-context dialogues to truncated segments, teaching it to generate coherent transitions even with incomplete input.
• Confidence-Based Selection: At inference, a confidence score (based on predictive entropy) determines whether a connective is safe to emit, preventing premature or nonsensical outputs.

3. Key Contributions

1. Problem Formulation: Identified the "response-onset dilemma" in cascaded systems and framed low-latency dialogue as a temporal reorganization problem rather than just a computation speed problem.
2. Novel Architecture: Introduced DDTSR, the first framework to explicitly decouple early connective generation from deep reasoning using small-large model synergy and cross-modal streaming.
3. Plug-and-Play Design: The framework is modular and compatible with diverse LLM backbones and existing ASR/TTS modules, requiring no retraining of the core large models.
4. Empirical Validation: Demonstrated significant latency reductions (19%–51%) while maintaining high discourse quality and coherence.

4. Experimental Results

The system was evaluated on two benchmarks: SD-Eval (daily conversation) and SpokenNativQA (question answering), using Qwen3 models (0.6B for connectives, 8B/32B for reasoning).

• Latency Reduction:
  • Waiting Latency: Reduced by 45.4% on SD-Eval (from 1003ms to 548ms) and 17.5% on SpokenNativQA.
  • Perception Latency: Dropped to ~100ms (compared to ~388ms in baselines) because the system starts processing before the user finishes speaking.
  • Reaction Latency: Reduced significantly by offloading the initial response generation to the lightweight model.
• Quality Preservation:
  • Text Coherence: DDTSR achieved consistency and coherence scores nearly identical to standard cascaded baselines (e.g., 4.77/4.49 vs. 4.76/4.54 on SD-Eval).
  • Speech Naturalness: UTMOSv2 scores remained comparable to baselines (~3.12 vs. 3.16), proving that stitching two audio tracks does not degrade perceptual quality.
• Scalability: Performance gains were robust across different model sizes (8B vs. 32B) and input lengths, with latency improvements becoming more pronounced for longer utterances.

5. Significance

• Human-Like Interaction: DDTSR bridges the gap between machine and human responsiveness, enabling "speak-while-thinking" behavior that mimics natural human turn-taking.
• Practicality: Unlike end-to-end speech models that require massive retraining and specialized hardware, DDTSR offers a modular, plug-and-play solution that can be integrated into existing cascaded pipelines to drastically improve user experience.
• Future Direction: It establishes a new paradigm for dialogue system design where discourse structure (connectives) is leveraged as an operational mechanism for latency reduction, rather than just a linguistic feature.

In conclusion, DDTSR successfully solves the latency-quality trade-off in spoken dialogue by leveraging the psychological insight that humans use low-risk discourse markers to maintain flow while planning complex responses.