Low-Latency Turn-Taking via Context-Aware Preface Generation in a Real-World Dialogue Robot

Technical Summary: Low-Latency Turn-Taking via Context-Aware Preface Generation

Problem Statement
Large Language Model (LLM)-based dialogue systems, particularly those utilizing cascaded ASR–LLM–TTS pipelines, suffer from significant response delays because generation typically begins only after the user's speech is fully recognized. While conversational fillers (e.g., "uh-huh," "I see") are commonly used to mask these delays, they often become unnatural over time if generic or repetitive. The core challenge lies in balancing response speed with generation quality: naive incremental generation can degrade consistency, yet waiting for full utterance recognition creates perceptible latency. Systems require a mechanism to hold the conversational floor with contextually appropriate preliminary responses while the substantive answer is being formulated.

Methodology
The authors propose a two-stage incremental response framework designed to decouple response preparation from speech onset. The system architecture consists of the following components:

1. Intent Readiness Detection: A binary classification model determines whether the user's intent has become sufficiently predictable from the current utterance prefix (conditioned on the preceding system utterance).

   • Training: The model was trained in two stages: first on an LLM-pseudo-labeled corpus (765k instances) and then fine-tuned on a human-annotated corpus (30k instances). Both datasets were diversified to include hesitations and self-repairs.
   • Architecture: Based on a Japanese ModernBERT 30M encoder with a binary classification head, utilizing focal loss to address class imbalance.
   • Trigger: When the intent-readiness score exceeds a threshold (0.35), the system triggers the generation of a short prefatory response.

2. Two-Stage Generation:

   • Stage 1 (Prefatory Response): Once intent readiness is detected, the system generates a short, context-aware prefatory response (e.g., "The burger shop..."). Strict constraints are applied: responses are limited to 10 Japanese characters, must not introduce new facts, make definitive statements, or express strong agreement.
   • Stage 2 (Main Response): Concurrently, the system generates the full, task-relevant main response based on the complete user utterance.

3. Turn-Taking Control: A Voice Activity Projection (VAP) model independently estimates when the user is yielding the turn. If a prefatory response is ready when the VAP predicts a turn transition, the system delivers it immediately. The main response follows once the full STT result is available.

Key Contributions

• Decoupled Architecture: The paper introduces a framework that separates the timing of speech onset (controlled by VAP) from the preparation of the response (controlled by Intent Readiness Detection).
• Context-Aware Prefaces: Unlike fixed fillers, the system generates preliminary responses conditioned on the user's emerging intent, bridging the gap between turn-taking prediction and substantive answer generation.
• Safety Constraints: The system enforces strict length and semantic constraints on prefatory responses to mitigate the risks of early prediction errors and hallucinations in real-world deployment.

Experimental Results
The framework was evaluated in a field experiment using a route-guidance robot in a Japanese shopping mall. Three conditions were compared: no-filler, fixed-filler (e.g., "yeah"), and contextual-preface.

• Initial Response Latency: Both the fixed-filler and contextual-preface conditions significantly reduced initial response latency compared to the no-filler baseline. However, the fixed-filler condition had a significantly shorter initial latency than the contextual-preface condition.
• Initial-to-Main Gap: The contextual-preface condition demonstrated a significantly shorter gap between the initial response and the main response compared to the fixed-filler condition. This indicates that while the preface takes slightly longer to generate than a fixed filler, it allows the main response to be delivered sooner relative to the start of the interaction.
• Trigger Timing: In the contextual-preface condition, the intent readiness detector fired on average after 5.53 characters (69.2% of the final utterance length), triggering before the user finished speaking in 58.2% of cases.
• Dialogue Breakdowns: Approximately 8% of prefatory responses were annotated as breakdowns (e.g., fragments or generic questions), primarily due to output truncation or weak contextual grounding.
• Subjective Ratings: Exploratory post-interaction questionnaires (30 per condition) showed no statistically significant differences across the four conditions regarding conversational pacing, naturalness, appropriateness, or satisfaction.

Significance and Claims
The paper claims that context-aware prefatory responses offer a practical trade-off for real-world dialogue robots. While they do not achieve the absolute lowest initial latency of fixed fillers, they significantly reduce the delay before the substantive answer is delivered, thereby improving the flow of information without relying on unnatural, repetitive fillers.

The authors modestly conclude that while the method enhances the practicality of response generation by filling post-turn waiting time, the lack of significant differences in subjective ratings suggests that larger, targeted studies are needed to determine the impact on overall user experience. The study highlights that stable, contextually grounded generation of short prefaces remains a key bottleneck, and future work must focus on improving the semantic and prosodic continuity between the prefatory and main responses.