🤖 AI

How Do Instructions Shape Speech? Cross-Attention Attribution for Style-Captioned Text-to-Speech

This paper introduces a novel cross-attention attribution method adapted for speech diffusion models to analyze how style-caption tokens influence acoustic outputs, revealing that style conditioning is most selective in early ODE steps and deep network layers while correlating strongly with fundamental frequency and energy.

Original authors: Nityanand Mathur, Hamees Sayed, Wasim Madha, Apoorv Singh, Sameer Khurana, Akshat Mandloi, Sudarshan Kamath

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nityanand Mathur, Hamees Sayed, Wasim Madha, Apoorv Singh, Sameer Khurana, Akshat Mandloi, Sudarshan Kamath

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magical voice actor who can sound like anyone, say anything, and speak in any mood you describe. You tell them, "Speak like a calm, deep, and slow robot," and they do it perfectly. But until now, nobody knew how the computer actually understood those words. Did the word "calm" change the whole voice? Did "robot" only affect the end of the sentence?

This paper is like putting on X-ray glasses to watch that magical voice actor work. The researchers built a tool to see exactly which words in your instructions influence the sound at every single moment of the speech.

Here is the breakdown of their discovery using simple analogies:

1. The Setup: The "DAAM" Glasses

In the world of image generation (like making pictures from text), scientists already had a way to see which words painted which parts of a picture. They called it "DAAM."

The researchers took this same idea and adapted it for speech. Since speech happens over time (like a movie) rather than in space (like a painting), they created "temporal heatmaps." Think of this as a timeline where they can see how much attention the computer pays to the word "loud" at every second of the audio.

2. The Big Discovery: The "Conductor" vs. The "Musicians"

The researchers looked at three types of words in your instructions:

Style words: Adjectives like "calm," "loud," or "deep."
Content words: Nouns like "voice," "speaker," or "male."
Function words: The boring glue like "a," "the," or "and."

The Finding:

Style words act like a Conductor. When the computer sees the word "calm," it pays attention to the entire song from start to finish. It doesn't just change one note; it sets the mood for the whole performance. The researchers found these words have very low "variance," meaning they spread their influence evenly across time, just like a conductor waving a baton over an entire orchestra.
Content words act like specific Musicians. Words like "male" or "female" are more focused. They influence the pitch and energy, but they are more localized to specific parts of the sound, similar to how a violinist plays a specific melody.
Function words are the sheet music. They are necessary for structure but don't really change the "vibe" of the voice.

3. The "Loud" Test: Do Words Match Reality?

The researchers wanted to know if the computer actually listened to the meaning of the words. They checked if the word "loud" made the computer pay more attention when the audio was actually loud.

The Result: Yes!

When the instruction said "loud," the computer's attention spiked exactly when the volume of the speech was high.
When it said "nervous," the computer paid attention when the energy of the voice was high.
When it said "confident," the computer focused on moments where the pitch (the highness/lowness of the voice) was higher.

This proves the computer isn't just guessing; it is genuinely connecting the meaning of the word to the actual sound it creates.

4. The Construction Site: When and Where Does the Magic Happen?

The computer builds the voice in steps, like a construction crew building a house. They looked at two things: Time Steps (when in the process) and Layers (how deep in the computer's brain).

The Early Steps (The Foundation): In the very beginning of the process, the computer is obsessed with the Style words. It's like laying the foundation of a house; it decides, "Okay, this whole house is going to be a castle." This is where the "global" mood is set.
The Deep Layers (The Details): As the process goes deeper, the computer starts focusing more on the Content words (like "male" or "female") to refine the specific identity of the voice.
The "Focus Point": Around the 17th layer of the computer's brain, something interesting happens. The computer becomes extremely focused. It stops looking at everything and zooms in on the most important words to make the final details perfect. It's like a photographer adjusting the lens to get the sharpest possible image right before taking the picture.

Summary

This paper is the first time we've been able to see the "thought process" of a text-to-speech AI. They found that:

Style words (like "calm") are the global directors, controlling the whole voice from start to finish.
The computer understands meaning: It links words like "loud" to actual loud sounds.
The process is a hierarchy: It starts by setting the big mood, then refines the specific identity, and finally sharpens the details at the very end.

Essentially, the computer doesn't just guess; it follows a very organized, step-by-step plan where different words have different jobs at different times to create the perfect voice.

Technical Summary: How Do Instructions Shape Speech? Cross-Attention Attribution for Style-Captioned Text-to-Speech

Problem Statement

Modern Text-to-Speech (TTS) systems have evolved from fixed speaker embeddings to natural language conditioning, where free-form style captions (e.g., "a calm, deep voice speaking slowly") control voice characteristics. While this paradigm offers expressive capabilities, a fundamental interpretability gap remains: how do individual words in a style caption influence the synthesized audio?

Unlike image generation, where Diffusion Attentive Attribution Maps (DAAM) have successfully mapped prompt tokens to spatial regions, speech presents unique challenges. Speech is inherently temporal, requiring attention to map tokens to spectrogram time frames rather than spatial pixels. Furthermore, the distinction between global properties (style, emotion) and local properties (phoneme identity, duration) creates an interpretability challenge specific to speech. Prior to this work, no equivalent framework existed to analyze cross-attention in speech diffusion models.

Methodology

The authors adapt the DAAM framework to the speech domain, applying it to CapSpeech-TTS, a non-autoregressive TTS model based on flow matching.

Model Architecture

The CapSpeech pipeline consists of:

Encoders: A T5 encoder processes the style caption into contextual embeddings ( $E_c$ ), while a CLAP encoder provides a global style tag embedding.
Generative Core: A Flow-Matching Diffusion Transformer (DiT) with 25 layers ( $L=25$ ). It iteratively refines mel-spectrogram latents from Gaussian noise over 24 ODE steps ( $S=24$ ).
Mechanism: At every layer and ODE step, the DiT computes multi-head cross-attention where audio queries attend to caption keys. This is the locus of style conditioning.
Vocoder: A HiFi-GAN vocoder converts the refined latent into the final waveform.

Attribution Mechanism

To analyze how captions influence generation, the authors:

Intercepted Attention: Registered forward hooks to capture cross-attention matrices ( $A^{(l,s)}_h$ ) at every layer $l$ and ODE step $s$ .
Aggregation: Averaged attention across heads, then aggregated across all 25 layers and 24 ODE steps to create a per-token temporal heatmap ( $M_j$ ) for each caption token $j$ .
Token Categorization: Classified caption tokens into three categories:
- Style: Adjectives describing quality/emotion (e.g., "calm," "loud").
- Content: Nouns describing the speaker (e.g., "voice," "male").
- Function: Articles, prepositions, punctuation.
Metrics: Defined metrics including temporal variance, peak-to-mean ratio (PMR), temporal entropy, and Pearson correlation with acoustic features (F0 and RMS energy).

Dataset

The study analyzed 3,600 combinations of 120 style captions and 30 diverse text transcripts, resulting in 3,520 successful generations. This yielded 54,880 token instances and over 2.1 million attention matrices.

Key Contributions

First Cross-Attention Attribution in TTS: The study provides the first per-token temporal heatmaps across all transformer layers and ODE steps for a flow-matching TTS model.
Global vs. Local Conditioning Analysis: Demonstrated that style tokens exhibit significantly lower temporal variance than content or function tokens, confirming their role as global modulators.
Acoustic Grounding: Showed that style-token attention correlates with measurable acoustic features (F0 and energy) in semantically coherent ways (e.g., "loud" correlates with energy).
Hierarchical Dynamics: Revealed that style conditioning peaks in early ODE steps and deep transformer layers, while attention entropy reaches a minimum at layer 17, coinciding with peak style importance.

Results

1. Global vs. Local Conditioning

Temporal Variance: Style tokens showed the lowest temporal variance ( $\bar{\sigma}^2 = 2.1 \times 10^{-5}$ ), significantly lower than content ( $7.0 \times 10^{-5}$ ) and function tokens ( $19.2 \times 10^{-5}$ ). This indicates style tokens distribute attention uniformly across the utterance.
Peak-to-Mean Ratio (PMR): Despite low variance, style tokens had the highest PMR (1.74), suggesting they produce compact, characteristic attention signatures with a distinctive peak spread uniformly over time.
Per-Word Hierarchy: Words describing global prosody (e.g., "cheerful," "deep") showed the most diffuse attention, while words with specific acoustic correlates (e.g., "loud," "nasal") showed higher variance, indicating partial localization to regions where their effect is strongest.

2. Acoustic Feature Correlations

Correlation Strength: Style tokens showed moderate positive correlations with F0 ( $\bar{r} = +0.21$ ) and energy ( $\bar{r} = +0.28$ ). Content tokens showed stronger correlations due to categorical constraints on pitch and energy.
Semantic Coherence: Specific words aligned with their acoustic definitions. "Loud" correlated strongly with energy ( $r = +0.64$ ), and "nervous" and "dramatic" correlated more with energy than F0. Conversely, "confident" correlated more with F0, consistent with pitch-raising in confident speech.

3. Layer and Step Dynamics

Layer Dynamics: Style importance increased monotonically through the network, peaking at Layer 17. The late-to-early importance ratio was 1.28. Content tokens peaked later (Layer 22), suggesting the network resolves style modulation first, then refines speaker identity. Function tokens showed flat importance across layers.
Step Dynamics: Style importance peaked at ODE Step 0 and decayed by a factor of 5.2× by the final step. This indicates that global acoustic scaffolding is established in the early stages of the diffusion process. Function tokens, conversely, increased in importance in later steps, shifting focus to fine-grained sequential structure.
Entropy: Attention entropy reached its minimum at Layer 18 (adjacent to the style importance peak), indicating maximal network selectivity at the most style-critical stage.

Significance and Claims

The paper claims to provide the first quantitative evidence that cross-attention in flow-matching TTS functions as a global modulation mechanism, fundamentally distinct from the temporal alignment role it plays in autoregressive TTS.

The study establishes that:

Style instructions are processed hierarchically: early ODE steps and deep transformer layers are critical for global style conditioning.
The network achieves a "coarse-to-fine" dynamic, establishing global structure first and refining acoustic details later.
Natural language instructions are not merely global tags but are functionally grounded in acoustic reality, with specific words driving specific acoustic features (F0, energy) in semantically coherent patterns.

This work bridges the gap between generative speech models and interpretability, offering a diagnostic tool for understanding failure modes and improving controllability in expressive TTS systems.

Note: The authors acknowledge limitations, noting the analysis is restricted to one model (CapSpeech) and synthetic prompts. Future work is suggested to extend to other architectures, natural user prompts, and causal interventions via attention editing.