Prosodic Boundary-Aware Streaming Generation for LLM-Based TTS with Streaming Text Input

This paper proposes a prosodic-boundary-aware post-training strategy for LLM-based TTS that enables natural streaming generation with incremental text input by learning early stopping at content boundaries and utilizing a sliding-window prompt to prevent long-form collapse, significantly outperforming existing baselines in both short and long-form scenarios.

The Gist

Imagine you are trying to tell a story to a friend, but you can only speak one sentence at a time, and you have to start speaking the moment the first sentence arrives. You don't know what the next sentence will be, but you need to sound natural, pause in the right places, and keep your voice consistent throughout the whole story.

This is the challenge of Streaming Text-to-Speech (TTS): turning text into speech in real-time as the text is being typed or generated.

The paper you shared introduces a clever new way to solve two major problems that happen when computers try to do this:

1. The "Robotic" Problem: Without seeing the future, the computer sounds unnatural. It doesn't know when to pause or change its tone because it's flying blind.
2. The "Memory Overload" Problem: If the story gets very long, the computer gets confused. It tries to remember everything it has ever said, gets overwhelmed, and starts hallucinating or making up nonsense words.

Here is how the authors fixed it, using some simple analogies:

1. The "Traffic Light" System (Prosodic Boundaries)

The Problem: Imagine driving a car where you can only see 5 meters ahead. You might speed up when you should be slowing down for a turn, or brake too late. Similarly, a TTS model needs to know where a sentence ends to know how to pause or change its emotion.

The Solution: The authors taught the AI to recognize a special "Traffic Light" (a Prosodic Boundary Marker).

• They trained the AI using a trick: they gave it a sentence, put a special invisible "stop sign" in the middle, and told it, "Okay, stop generating audio right here."
• This taught the AI that even if it doesn't know the whole story yet, it knows exactly where the current "chunk" of the story ends. It learns to pause naturally at these signs, just like a human speaker would.

2. The "Sliding Window" vs. The "Infinite Backpack"

The Problem:

• Old Method (The Infinite Backpack): Imagine a student trying to write an essay. They keep every single word they've ever written in a giant backpack. As the essay gets longer, the backpack gets so heavy they can't move, and they start dropping things or forgetting what they wrote earlier. In AI terms, this is "unbounded context," which causes the model to crash or sound garbled after a few minutes.
• The New Method (The Sliding Window): Instead of carrying the whole backpack, imagine the student only keeps the last few pages of their essay in their hand. As they write a new page, they toss the oldest page away.

The Solution: The authors use a Sliding Window.

• The AI only looks at the current chunk of text (e.g., 5 words) plus a tiny bit of "future" text (e.g., 2 words) to plan its tone.
• Once it finishes that chunk, it slides the window forward. It remembers the sound of the last chunk to keep its voice consistent, but it forgets the specific words of the distant past to stay light and fast.

3. The "Seamless Stitch" (Acoustic Prompting)

The Problem: If you stitch two pieces of fabric together without care, you get a rough seam. If you stitch two chunks of speech together, you might hear a weird click or a sudden change in pitch.

The Solution: The AI uses the very last sound of the previous chunk as a "primer" for the next chunk. It's like a singer humming the last note of a phrase to help them start the next phrase smoothly. This ensures the voice sounds like one continuous person, not a robot switching voices every few seconds.

The Results: Why It Matters

The researchers tested this on a "long-form" task (reading a whole paragraph).

• The Old Way: The AI got confused, started making up words, and the error rate skyrocketed (it was wrong 71% of the time!).
• The New Way: The AI stayed calm, kept its voice consistent, and only made mistakes 4.8% of the time.

In short: This paper teaches an AI how to tell a long story in real-time without getting a headache. It does this by giving the AI "traffic lights" to know when to pause, a "sliding window" to keep its memory light, and a "seamless stitch" to keep its voice smooth. This makes voice assistants and live translation tools sound much more human and reliable.

Technical Summary

Here is a detailed technical summary of the paper "Prosodic Boundary-Aware Streaming Generation for LLM-Based TTS with Streaming Text Input."

1. Problem Statement

The paper addresses two critical challenges in Streaming Text-to-Speech (TTS) systems that process streaming text input (where audio is generated as text arrives in real-time):

1. Unnatural Prosody: High-quality speech synthesis requires sufficient context, including future text (lookahead), to predict prosodic features like stress, pauses, and intonation. Streaming systems with restricted receptive fields lack this lookahead, leading to flat or unnatural prosody. Existing solutions often require complex causal attention modifications or precise text-speech forced alignment, which are difficult to implement.
2. Long-Form Generation Collapse: Modern LLM-based TTS models (e.g., CosyVoice) often use interleaved text and speech tokens. In long-form streaming scenarios, the physical distance between a text token and its corresponding speech tokens grows unbounded. This causes the model's context to become unstable, leading to semantic drift, hallucinations, and catastrophic generation failure (e.g., garbled speech or premature termination).

2. Methodology

The authors propose a prosodic-boundary-aware post-training strategy that adapts pre-trained LLM-based TTS models using only weakly time-aligned data (word-level timestamps) without modifying the underlying architecture.

Key Components:

• Prosodic-Boundary Marker:
  • A special token (marker_boundary) is inserted into the text sequence to act as a "soft boundary."
  • During training, the model learns to treat this marker as a segmentation cue and a prosodic anchor, allowing it to plan intonation based on limited future context up to the boundary.
• Training with Weakly Time-Aligned Supervision:
  • Data Preparation: Word-level timestamps are extracted using an off-the-shelf aligner (WhisperX).
  • Dynamic Boundary Insertion: During fine-tuning, the model is trained on truncated sequences. With probability $p_{full}$, the full utterance is used. Otherwise, a random word index is selected, the marker_boundary is inserted, and the target speech sequence is truncated to the aligned audio end of that word. This forces the model to learn early stopping and prosodic planning within bounded segments.
• Bounded Context & Sliding-Window Continuation:
  • Inference Pipeline: Input text is processed in chunks of $k$ words with a lookahead of $f$ future words.
  • Sliding-Window Prompt: To maintain continuity across chunks, the prompt for the current chunk includes the text and speech tokens generated in the previous step.
  • Acoustic Prompting: The audio tail of the previous chunk is used as an acoustic prompt to ensure seamless concatenation.
  • Complexity: This design keeps the Key-Value (KV) cache bounded at $O(k + f)$, preventing memory growth and instability regardless of the total sequence length.

3. Key Contributions

1. Prosodic-Boundary Adaptation: Introduced a novel adaptation method combining boundary markers with a windowed lookahead mechanism. This improves prosody without requiring complex causal architectural changes or precise forced alignment.
2. Acoustic Prompting for Continuity: Designed a method utilizing the previous chunk's audio tail to ensure seamless concatenation and mitigate generation collapse in long-form cross-modality streaming.
3. Robust Long-Form Streaming: Demonstrated that robust, stable long-form streaming can be achieved using only weakly time-aligned open-source data, significantly outperforming existing interleaved baselines in real-time deployment.

4. Experimental Results

The method was evaluated on the Seed-TTS-Eval benchmark (standard short sentences) and an LLM-expanded Long-form benchmark (280–320 word paragraphs).

• Streaming Efficiency:
  • The proposed method achieved the lowest Time-to-First-Audio (TTFA) of 1296 ms, outperforming both Interleaved (1414 ms) and Sliding-Window (2588 ms) baselines.
  • It achieved a Real-Time Factor (RTF) of 0.782 (using streaming vocoding), which is more efficient than the Interleaved baseline (0.843).
• Objective Quality (Long-Form):
  • Word Error Rate (WER): The proposed method reduced WER from 71.0% (Interleaved baseline) to 4.8%, a 66.2% absolute reduction. This indicates the model successfully avoided the catastrophic failure seen in baselines.
  • Speaker Similarity (SPK-SIM): Increased by 16.1% relative to the baseline in long-form scenarios (0.65 vs. 0.56).
  • Emotion Similarity (EMO-SIM): Increased by 1.5% relative (0.912 vs. 0.899).
• Subjective Quality (MOS):
  • The proposed method achieved the highest Mean Opinion Scores (MOS) across all metrics.
  • In long-form scenarios, it maintained a MOS of 4.13, compared to 3.18 for the Interleaved baseline and 1.60 for the Sliding-Window baseline.
  • It successfully preserved speaker identity and emotional consistency where other methods failed.

5. Significance

This work provides a robust, architecture-agnostic solution for streaming TTS with incremental text input. By decoupling acoustic generation spans from the broader prosodic context using simple boundary markers, the authors solved the "long-form collapse" problem inherent in interleaved LLM-TTS models.

The significance lies in:

• Practicality: It enables high-quality, low-latency streaming TTS for interactive systems (e.g., dialogue agents, speech-to-speech translation) without needing precise alignment data or heavy model retraining.
• Stability: It proves that bounded context strategies can effectively prevent the semantic drift and hallucinations that plague long-form generation in large language models applied to speech.
• Efficiency: It achieves state-of-the-art performance with a computationally efficient sliding-window approach, making it viable for real-time deployment on standard hardware.