💬 NLP

Style Amnesia: Investigating Speaking Style Degradation and Mitigation in Multi-Turn Spoken Language Models

This paper identifies and investigates "style amnesia" in spoken language models, demonstrating that current systems fail to consistently maintain paralinguistic styles (such as emotion, accent, volume, and speed) over multi-turn conversations despite explicit instructions, though performance can be partially mitigated through explicit style recall.

Original authors: Yu-Xiang Lin, Cheng-Han Chiang, Hung-yi Lee

Published 2026-04-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yu-Xiang Lin, Cheng-Han Chiang, Hung-yi Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: The "Forgetful Actor"

Imagine you hire a talented actor to play a specific character in a play. You give them a very clear instruction before the show starts: "Throughout the entire play, you must speak in a deep, grumpy voice and sound very angry."

In the first scene, the actor is perfect. They are growling, stomping their feet, and sounding furious. But as the play moves to the second scene, third scene, and fourth scene, something strange happens. The actor slowly forgets they are supposed to be angry. By the final scene, they are back to speaking in their normal, cheerful, polite voice.

This is exactly what the researchers found happening with Spoken Language Models (SLMs)—the AI voices you talk to on your phone or computer. They call this phenomenon "Style Amnesia."

What They Discovered

The researchers tested several famous AI voice models (like GPT-4o, Gemini Live, and open-source models) to see if they could keep a specific "style" going during a long conversation. They asked the AIs to:

Speak with a specific emotion (like sadness or anger).
Use a specific accent (like an Indian or American accent).
Speak at a specific volume (loud or quiet).
Speak at a specific speed (fast or slow).

The Results:

Great Start, Bad Finish: Every single model was great at following the instructions in the very first turn. But as the conversation continued, they quickly forgot the rule.
The "System Message" Trap: The researchers tried putting the instruction in the "System Message" (a special hidden note meant for the AI's permanent memory, like a rulebook). Surprisingly, this made things worse. The AIs ignored the hidden rulebook and listened better when the instruction was spoken out loud by the "user" in the chat.
They Remember, But Can't Do: When the researchers stopped the conversation and asked, "What style were you supposed to use?" the AIs could answer correctly! They hadn't actually forgotten the instruction; they just couldn't perform it while talking. It's like an actor who knows the script perfectly but keeps breaking character when the camera starts rolling.

Why Does This Happen? (The "Attention" Analogy)

To understand why, imagine the AI's brain as a spotlight.

Turn 1: The spotlight is shining brightly on the instruction: "Speak Sadly." The AI focuses 100% on that.
Turn 4: As the conversation gets longer, the spotlight gets dimmer and starts wandering around the room. The instruction is still there in the background, but the AI is now focusing more on the new words the user just said. The instruction gets "diluted" or drowned out by the noise of the conversation.

The Solution: The "Memory Jog"

Since the AI isn't actually forgetting the rule, the researchers tried a simple fix. After every turn, they asked the AI a quick question before it answered: "Remember, you are supposed to speak [sadly/fast/loudly]. Now, please answer the user."

The Result: This worked like a charm! By forcing the AI to "recall" the rule right before speaking, the "forgetfulness" went away. The AI stayed in character much longer.

Why This Matters

We are moving toward a future where we talk to AI assistants for hours, not just one question at a time. If you ask an AI to be your "grumpy but helpful mechanic," you want it to stay grumpy and helpful the whole time, not turn into a cheerful robot halfway through.

This paper tells us that current AI voices are like actors with short attention spans. They need a little nudge (a "memory jog") to stay in character, and we need to build better systems that can hold onto their personality without needing constant reminders.

Summary in One Sentence

Current AI voices are great at following style instructions at the start of a chat, but they quickly "forget" to keep that style as the conversation goes on, unless we explicitly remind them to do so.

1. Problem Definition

The paper addresses a critical gap in the evaluation of Spoken Language Models (SLMs). While current SLMs can generate speech with specific paralinguistic attributes (e.g., emotion, accent, volume, speed) in single-turn interactions, their ability to maintain these styles over multi-turn conversations remains unexplored.

The authors identify a phenomenon they term "Style Amnesia": the tendency of SLMs to adhere to a speaking style instruction in the first turn but rapidly degrade in adherence as the conversation progresses. This is distinct from text-based LLMs losing context; here, the model may "remember" the instruction but fails to execute the acoustic or stylistic constraints in its generated speech.

2. Methodology

2.1 Evaluation Framework

The authors constructed a multi-turn evaluation framework involving:

Models Evaluated: Three proprietary models (GPT-4o, GPT-4o mini, Gemini Live) and two open-source end-to-end SLMs (Qwen2.5-Omni, Step-Audio 2 mini). A cascaded baseline (ASR + Text LLM + TTS) was also used as an upper-bound reference.
Interaction Setup: A User Simulator (a cascade of ASR, a text-only LLM, and a TTS) interacts with the evaluated SLM. The simulator initiates conversations using 100 diverse topics from the Soda dataset.
Style Instructions: At the start of the dialogue, the user instructs the SLM to maintain a specific paralinguistic style throughout the conversation. The styles tested include:
- Emotion: Sad, Happy, Angry, Neutral.
- Accent: North American, Indian.
- Volume: Loud, Quiet.
- Speed: Fast, Slow.
Metrics:
- Instruction-Following (IF) Rate ( $IF_j$ ): The percentage of turns where the generated speech matches the target style, determined by automatic judges.
- Degradation Rate ( $D$ ): The average absolute drop in IF rate from Turn 1 to subsequent turns ( $K=4$ ).

2.2 Automatic Judges

To quantify style adherence without human intervention for every sample, the authors employed specific models:

Emotion: Emotion2vec-Large (predicts probability of happiness, sadness, anger, neutral).
Accent: Voxlect-English-Dialect-Whisper-Large-v3 (predicts North American vs. Indian).
Volume: PyLoudnorm (measures LUFS relative to a neutral baseline).
Speed: Parakeet TDT v2 (measures Words Per Minute relative to a baseline).
Validation: Human annotators (via MTurk) and Gemini-2.5 Pro were used to validate the automatic judges, showing high correlation (Cohen's Kappa $\approx$ 0.8 for accent, $\approx$ 0.47 for emotion).

3. Key Findings & Results

3.1 Existence of Style Amnesia

Rapid Degradation: All evaluated SLMs showed a significant drop in IF rates after the first turn. For example, GPT-4o mini's adherence to "Sadness" dropped from a high initial rate to a degradation rate of 65.3%.
Universal Issue: The phenomenon was observed across all model architectures (proprietary, open-source, cascaded) and all style categories, though the severity varied.
Default Bias: Models performed significantly better on "default" styles (Happy/Neutral tone, North American accent) because these often align with their pre-trained baselines, resulting in artificially low degradation rates for these specific styles.

3.2 Root Cause Analysis

Attention Dilution: By analyzing the attention weights of the open-source Step-Audio 2 mini, the authors found that attention allocated to the initial style instruction tokens dropped from ~8% in Turn 1 to <0.6% by Turn 4. This suggests the model loses focus on global constraints as the context window fills with dialogue history.
Prompt Placement: Contrary to the intuition that "System Messages" provide persistent instructions, placing style instructions in system prompts resulted in worse performance than placing them in the first user message. Models often ignored system-level style constraints entirely.
Memory vs. Execution: A crucial finding is that models do not forget the instruction. When explicitly prompted to "recall" the style, proprietary models achieved near-perfect recall rates (e.g., 100%). However, they still failed to execute the style in the subsequent speech generation. This indicates the failure is in stylistic control, not memory retention.

3.3 Text-Acoustic Synergy

Semantic vs. Acoustic: For emotion styles, both semantic content and acoustic features degraded simultaneously.
Speed Strategies: Models used different strategies for speed. Some (Gemini Live) reduced word count to speak faster, while others (GPT-4o) compressed the same amount of text into shorter durations. Over turns, the distinction between "Fast" and "Slow" speech rates narrowed, indicating a loss of control over the acoustic parameter.

4. Mitigation Strategy

The authors proposed a Recall Process to mitigate style amnesia:

Mechanism: Before generating a response in turns 2–4, the system prompts the SLM to explicitly state the required speaking style (e.g., "What speaking style should you use?").
Results: This explicit recall significantly reduced the degradation rate.
- For GPT-4o mini, the degradation rate for "Sadness" dropped from 65.3% to 30.3% (a ~35% improvement).
- For "Indian Accent," GPT-4o mini improved from 49.7% degradation to 14.9%.
Implication: While the model struggles to sustain the style autonomously, it can successfully re-apply the style when the constraint is re-activated in the immediate context.

5. Significance and Contributions

New Benchmark: The paper introduces the first systematic evaluation of multi-turn speaking style consistency, highlighting that current SLMs are not yet ready for long-form, style-consistent interactions (e.g., role-playing, storytelling).
Diagnosis of Failure: It distinguishes between memory loss (forgetting the instruction) and execution failure (inability to generate the style), identifying the latter as the primary bottleneck.
Architectural Insight: The finding that system prompts are less effective than user prompts for style control challenges current prompt engineering paradigms for SLMs.
Future Direction: The paper argues that future SLMs need mechanisms to "anchor" global style constraints throughout the generation process, rather than relying on the initial context window alone. The proposed "Recall Process" offers a practical, immediate mitigation strategy.

Conclusion

The paper concludes that Style Amnesia is a fundamental limitation in current SLMs. While models can recall instructions when prompted, they fail to maintain stylistic consistency autonomously over multiple turns. This gap between style retention and stylistic expression represents a major hurdle for deploying SLMs in applications requiring long-term persona or emotional consistency.