💬 NLP

Knowing What to Stress: A Discourse-Conditioned Text-to-Speech Benchmark

This paper introduces Context-Aware Stress TTS (CAST), a new benchmark demonstrating that while language models can infer contextually appropriate word stress from discourse, current text-to-speech systems frequently fail to accurately realize this emphasis in synthesized speech.

Original authors: Arnon Turetzky, Avihu Dekel, Hagai Aronowitz, Ron Hoory, Yossi Adi

Published 2026-04-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Arnon Turetzky, Avihu Dekel, Hagai Aronowitz, Ron Hoory, Yossi Adi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are telling a story to a friend. You say the exact same sentence: "The manager booked the flight."

But depending on what you were talking about before that sentence, the meaning changes completely:

If your friend thought the assistant did it, you stress the word Manager to say, "No, it was the Manager!"
If your friend thought they booked a hotel, you stress the word Flight to say, "No, it was the flight!"

In human speech, we do this naturally. We shift our voice to highlight the right word. But can a computer do this?

This paper introduces a new test called CAST (Context-Aware Stress TTS) to find out. Here is the breakdown in simple terms:

1. The Problem: The Robot's "Flat" Voice

Modern Text-to-Speech (TTS) systems (like the voices in your GPS or Siri) are getting very good at sounding human. They can whisper, shout, and sound happy or sad.

However, the researchers found a major blind spot: Robots don't know why they should stress a word. They can read the sentence, but they can't listen to the conversation before the sentence to figure out which word needs the spotlight.

2. The Test: The "Same Sentence, Different Story" Game

The researchers built a benchmark (a test suite) called CAST. Think of it like a "spot the difference" game for audio.

The Setup: They take one sentence (e.g., "She chose the cello").
The Twist: They pair it with two different backstories.
- Story A: "I thought she picked the violin." (So the robot should stress Cello).
- Story B: "I thought she played the piano." (So the robot should stress Cello... wait, let's try a better example from the paper).
- Better Example: "The manager booked the flight."
  - Context A: "I thought the assistant did it." -> Robot must stress Manager.
  - Context B: "I thought they booked a hotel." -> Robot must stress Flight.

The robot has to listen to the context, figure out the story, and then speak the sentence with the right word emphasized.

3. The Results: The "Brain vs. Mouth" Gap

The researchers tested the smartest AI systems available. Here is what they found, using a simple analogy:

The Brain (Text-Only AI): If you ask a text-based AI (like a smart chatbot) "Which word should I stress here?", it gets it right 88% of the time. It understands the story perfectly. It knows the logic.
The Mouth (Speech AI): When you ask the same AI to speak the sentence with that stress, it fails miserably. It often stresses the wrong word, or stresses the same word every time regardless of the story.

The Analogy: Imagine a brilliant actor who can read a script and understand exactly where the emotion lies (the Text AI). But when they try to perform it on stage, they forget to change their tone, delivering the lines in a flat, robotic monotone (the TTS system). The brain knows the joke; the mouth just tells the punchline without the laugh.

4. Why This Matters

Currently, if you ask a robot to tell a story, it might sound smooth, but it won't sound smart. It won't correct your mistakes or clarify your confusion naturally because it doesn't know which word to highlight.

The researchers also found that even if you tell the robot explicitly, "Hey, stress the word 'Manager'," it still struggles to do it perfectly. This means the technology for "emotional" or "context-aware" speech is still in its toddler years.

5. What They Did Next

To help fix this, the team didn't just point out the problem; they built the tools to solve it:

The Test: They released the CAST benchmark so other scientists can test their robots.
The Data: They generated 10,000 examples of "Context + Sentence + Audio" to train future robots.
The Pipeline: They made the code public so anyone can build better tests.

The Bottom Line

We have built AI that can read a story and understand the subtext. But we haven't yet built AI that can speak that subtext naturally. The "brain" is ready, but the "voice" is still learning how to listen. This paper is a roadmap to help the voice catch up to the brain.

1. Problem Statement

While modern Text-to-Speech (TTS) systems have achieved high expressiveness, it remains unclear whether they can infer contextually appropriate word-level stress from discourse alone.

The Core Issue: Spoken meaning often relies on prosody (emphasis) to convey contrast, correction, or clarification. The same sentence (e.g., "The manager booked the flight") can have different meanings depending on which word is stressed (e.g., manager vs. flight), based on the preceding context.
The Gap: Current TTS systems often fail to shift stress dynamically based on discourse context. They tend to rely on sentence-internal biases or default patterns rather than inferring the intended focus from the surrounding conversation. Existing benchmarks either focus on stress detection from audio, require explicit stress markers, or evaluate text-level understanding without speech synthesis.

2. Methodology: The CAST Benchmark

The authors introduce CAST (Context-Aware Stress TTS), a benchmark designed to evaluate end-to-end TTS systems on their ability to infer and realize context-dependent stress.

A. Task Definition

Input: A textual context ( $c$ ) and a target sentence ( $s$ ).
Output: Synthesized speech ( $\hat{y}$ ).
Constraint: No explicit stress annotations or emphasis markers are provided at inference time.
Goal: The system must infer the intended stressed word ( $w^*$ ) based on $c$ and $s$ , and acoustically realize that emphasis.

B. Benchmark Construction Pipeline

Generative Design: Using structured prompting (with gpt-5-mini), the authors generate contrastive context pairs.
- Structure: Identical target sentences are paired with two distinct contexts (Context A and Context B) that require stressing different words.
- Example: Sentence: "She published the essay in the winter issue."
  - Context A (Correction): "The editor wrote it ran in the spring." $\rightarrow$ Stress winter.
  - Context B (Clarification): "I believed she sent a draft." $\rightarrow$ Stress published.
Validation: Items are filtered through a multi-judge consistency check using two independent LLMs (gpt-5-nano and gemini-2.5-flash). Items where judges disagree on the intended stress are discarded.
Dataset: The final evaluation set contains 113 contrastive pairs (226 items), balanced across stress positions (Initial, Early, Medial, Final) and pragmatic phenomena (Correction, Exclusivity, Role disambiguation, etc.).
Synthetic Corpus: An extended pipeline was used to generate a separate training corpus of ~10,000 context-sentence-audio triples for future research.

C. Evaluation Protocol

Systems Evaluated: A diverse set including non-LLM (Kokoro), LLM-based (Chatterbox, CosyVoice, HiggsAudio V2, Qwen3-TTS, GPT-4o-mini-tts).
Conditioning Modes:
1. Baseline: No context.
2. Concat: Context prepended to the sentence.
3. Instruct: Context provided as a natural language instruction.
4. Explicit: Target word specified via markup (Oracle upper bound).
Metrics:
- Hit: Did the system stress the target word?
- Pair-Contrast: Did the system stress the target word and avoid stressing the alternative word (required for the opposing context)? This penalizes over-expressive stress.
- Pair-Correct: The strictest metric. Requires the system to correctly handle both sides of the contrastive pair simultaneously (i.e., no sentence-internal bias).
Stress Detection: The authors use WHISTRESS, an automatic stress detection model fine-tuned on Whisper, validated against human annotators.

3. Key Contributions

CAST Benchmark: The first benchmark specifically designed to test context-conditioned word-level stress in TTS using contrastive context pairs with identical lexical content.
Scalable Pipeline: A released construction pipeline and a synthetic corpus (~10k samples) to support the training of context-aware TTS systems.
Systematic Evaluation: A comprehensive study revealing a significant performance gap between text-level inference and speech-level realization.

4. Key Results

A. TTS Performance (Speech Generation)

Failure to Adapt: All evaluated TTS systems showed near-zero Pair-Correct scores across all conditioning modes (Concat, Instruct, Baseline).
Context Ignorance: Providing context (via concatenation or instructions) yielded no statistically significant improvement over the baseline. The confidence intervals for Pair-Contrast and Pair-Correct overlapped substantially across all context conditions.
Oracle Gap: Even when the target word was explicitly specified (Explicit mode), performance improved (e.g., CosyVoice Pair-Correct rose to ~10.6%), but remained far from perfect. This suggests that while the capacity for stress realization exists, it is not reliably triggered by context alone.

B. Text-Only Inference (Language Models)

High Semantic Understanding: When evaluated as text-only models (predicting the stressed word directly from text), models performed significantly better.
Scale Dependency: Performance correlated with model size. Claude-haiku-4.5 achieved 88.1% Contrast and 76.1% Correct.
Implication: The intended stress is recoverable from discourse context at the text level. The failure lies in the acoustic realization step within TTS systems, not in the semantic understanding of the context.

5. Significance and Conclusion

The "Stress Gap": The paper establishes a critical disconnect: TTS systems can understand what to say, but they fail to know how to say it based on context. They default to static, sentence-internal stress patterns rather than dynamic, discourse-driven prosody.
Future Directions: The work highlights that improving TTS requires more than just better acoustic models; it necessitates architectures that can effectively map discourse context to prosodic control signals.
Resource Availability: The authors release the benchmark, evaluation framework, construction pipeline, and a synthetic training corpus to accelerate research in context-aware speech synthesis.

In summary, CAST demonstrates that while modern LLMs can semantically infer the correct emphasis from context, current TTS systems fail to translate this inference into the acoustic reality of speech, marking a major open challenge for the field of expressive speech synthesis.